2020-03-11T22:21:26.677 INFO:root:teuthology version: 1.0.0-1def34b 2020-03-11T22:21:26.678 DEBUG:teuthology.run:Teuthology command: teuthology /tmp/teuthology-worker.VM2TZR.tmp -- --verbose --lock --description rados/standalone/{supported-random-distro$/{centos_8.yaml} workloads/osd.yaml} --name dzafman-2020-03-11_15:13:33-rados-wip-zafman-octo-diag-distro-basic-smithi --block --owner scheduled_dzafman@bionic-laptop --archive /home/teuthworker/archive/dzafman-2020-03-11_15:13:33-rados-wip-zafman-octo-diag-distro-basic-smithi/4848256 2020-03-11T22:21:26.714 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-03-11T22:21:26.772 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/dzafman-2020-03-11_15:13:33-rados-wip-zafman-octo-diag-distro-basic-smithi/4848256 branch: wip-zafman-octo-diag description: rados/standalone/{supported-random-distro$/{centos_8.yaml} workloads/osd.yaml} email: dzafman@redhat.com first_in_suite: false job_id: '4848256' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: dzafman-2020-03-11_15:13:33-rados-wip-zafman-octo-diag-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: centos os_version: '8.1' overrides: admin_socket: branch: wip-zafman-octo-diag ceph: conf: mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug filestore: 20 debug journal: 20 debug ms: 1 debug osd: 25 log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 4acc1121e47039a3cdf2978ae458c10d153aa141 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: 4acc1121e47039a3cdf2978ae458c10d153aa141 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-zafman-octo-diag sha1: 4acc1121e47039a3cdf2978ae458c10d153aa141 owner: scheduled_dzafman@bionic-laptop priority: 20 repo: git://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 sha1: 4acc1121e47039a3cdf2978ae458c10d153aa141 suite: rados suite_branch: wip-zafman-octo-diag suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_wip-zafman-octo-diag/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: 4acc1121e47039a3cdf2978ae458c10d153aa141 tasks: - install: null - workunit: basedir: qa/standalone clients: all: - osd teuthology_branch: master tube: smithi verbose: false worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.27440 2020-03-11T22:21:26.773 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_wip-zafman-octo-diag/qa; will attempt to use it 2020-03-11T22:21:26.773 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_wip-zafman-octo-diag/qa/tasks 2020-03-11T22:21:26.773 INFO:teuthology.run_tasks:Running task internal.check_packages... 2020-03-11T22:21:26.775 INFO:teuthology.task.internal:Checking packages... 2020-03-11T22:21:26.796 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'basic' and ceph hash '4acc1121e47039a3cdf2978ae458c10d153aa141' 2020-03-11T22:21:26.796 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2020-03-11T22:21:26.796 INFO:teuthology.packaging:ref: None 2020-03-11T22:21:26.797 INFO:teuthology.packaging:tag: None 2020-03-11T22:21:26.797 INFO:teuthology.packaging:branch: wip-zafman-octo-diag 2020-03-11T22:21:26.797 INFO:teuthology.packaging:sha1: 4acc1121e47039a3cdf2978ae458c10d153aa141 2020-03-11T22:21:26.797 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-zafman-octo-diag 2020-03-11T22:21:27.033 INFO:teuthology.task.internal:Found packages for ceph version 15.1.0-1941.g4acc112 2020-03-11T22:21:27.034 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2020-03-11T22:21:27.048 INFO:teuthology.task.internal:no buildpackages task found 2020-03-11T22:21:27.048 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2020-03-11T22:21:27.061 INFO:teuthology.task.internal.lock_machines:Locking machines... 2020-03-11T22:21:27.062 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-03-11T22:21:27.116 DEBUG:teuthology.lock.ops:lock_many request: {'count': 1, 'locked_by': 'scheduled_dzafman@bionic-laptop', 'description': '/home/teuthworker/archive/dzafman-2020-03-11_15:13:33-rados-wip-zafman-octo-diag-distro-basic-smithi/4848256', 'machine_type': 'smithi'} 2020-03-11T22:21:27.134 DEBUG:teuthology.lock.ops:locked ubuntu@smithi059.front.sepia.ceph.com 2020-03-11T22:21:27.164 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=, signals=[15]) 2020-03-11T22:21:27.362 INFO:teuthology.provision.fog.smithi059:Scheduling deploy of centos 8.1 2020-03-11T22:21:27.769 INFO:teuthology.orchestra.console:Power off smithi059 2020-03-11T22:21:27.769 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-03-11T22:21:27.795 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-11T22:21:31.823 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-11T22:21:35.942 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-11T22:21:36.152 INFO:teuthology.orchestra.console:Power off for smithi059 completed 2020-03-11T22:21:36.253 INFO:teuthology.orchestra.console:Power on smithi059 2020-03-11T22:21:36.253 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-03-11T22:21:36.274 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-11T22:21:40.299 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-11T22:21:44.414 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-11T22:21:44.632 INFO:teuthology.orchestra.console:Power on for smithi059 completed 2020-03-11T22:21:44.733 INFO:teuthology.provision.fog.smithi059:Waiting for deploy to finish 2020-03-11T22:24:15.862 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi059.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:24:24.923 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi059.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:24:33.978 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi059.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:24:43.034 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi059.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:24:52.091 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi059.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:25:01.147 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi059.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:26:07.150 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi059.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:26:07.403 INFO:teuthology.orchestra.run.smithi059:> hostname 2020-03-11T22:26:07.707 INFO:teuthology.orchestra.run.smithi059.stdout:smithi059.front.sepia.ceph.com 2020-03-11T22:26:07.708 INFO:teuthology.orchestra.run.smithi059:> grep smithi059.front.sepia.ceph.com /etc/hosts 2020-03-11T22:26:07.723 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.59 smithi059.front.sepia.ceph.com smithi059 2020-03-11T22:26:07.724 INFO:teuthology.orchestra.run.smithi059:> sudo hostname smithi059 2020-03-11T22:26:07.852 INFO:teuthology.orchestra.run.smithi059:> sudo sed -i -e 's/smithi059.front.sepia.ceph.com/smithi059/g' /etc/hosts 2020-03-11T22:26:07.884 INFO:teuthology.orchestra.run.smithi059:> sudo sed -i -e 's/172.21.15.59/172.21.15.59/g' /etc/hosts 2020-03-11T22:26:07.955 INFO:teuthology.orchestra.run.smithi059:> sudo sed -i -e 's/smithi059.front.sepia.ceph.com/smithi059/g' /etc/hostname 2020-03-11T22:26:08.026 INFO:teuthology.orchestra.run.smithi059:> sudo hostname smithi059 2020-03-11T22:26:08.096 INFO:teuthology.provision.fog.smithi059:Deploy complete! 2020-03-11T22:26:08.098 DEBUG:teuthology.parallel:result is None 2020-03-11T22:26:08.204 INFO:teuthology.lock.ops:Checking smithi059.front.sepia.ceph.com 2020-03-11T22:26:08.204 INFO:teuthology.lock.ops:New key found. Updating... 2020-03-11T22:26:08.225 INFO:teuthology.lock.ops:Updating [smithi059.front.sepia.ceph.com]: set os type and version on server 2020-03-11T22:26:08.225 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-11T22:26:08.226 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi059.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:26:08.295 INFO:teuthology.orchestra.run.smithi059:> true 2020-03-11T22:26:08.476 INFO:teuthology.orchestra.run.smithi059:> uname -m 2020-03-11T22:26:08.530 INFO:teuthology.orchestra.run.smithi059.stdout:x86_64 2020-03-11T22:26:08.531 INFO:teuthology.orchestra.run.smithi059:> cat /etc/os-release 2020-03-11T22:26:08.585 INFO:teuthology.orchestra.run.smithi059.stdout:NAME="CentOS Linux" 2020-03-11T22:26:08.585 INFO:teuthology.orchestra.run.smithi059.stdout:VERSION="8 (Core)" 2020-03-11T22:26:08.586 INFO:teuthology.orchestra.run.smithi059.stdout:ID="centos" 2020-03-11T22:26:08.586 INFO:teuthology.orchestra.run.smithi059.stdout:ID_LIKE="rhel fedora" 2020-03-11T22:26:08.586 INFO:teuthology.orchestra.run.smithi059.stdout:VERSION_ID="8" 2020-03-11T22:26:08.586 INFO:teuthology.orchestra.run.smithi059.stdout:PLATFORM_ID="platform:el8" 2020-03-11T22:26:08.587 INFO:teuthology.orchestra.run.smithi059.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2020-03-11T22:26:08.587 INFO:teuthology.orchestra.run.smithi059.stdout:ANSI_COLOR="0;31" 2020-03-11T22:26:08.587 INFO:teuthology.orchestra.run.smithi059.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2020-03-11T22:26:08.587 INFO:teuthology.orchestra.run.smithi059.stdout:HOME_URL="https://www.centos.org/" 2020-03-11T22:26:08.587 INFO:teuthology.orchestra.run.smithi059.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-03-11T22:26:08.588 INFO:teuthology.orchestra.run.smithi059.stdout: 2020-03-11T22:26:08.588 INFO:teuthology.orchestra.run.smithi059.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2020-03-11T22:26:08.589 INFO:teuthology.orchestra.run.smithi059.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2020-03-11T22:26:08.589 INFO:teuthology.orchestra.run.smithi059.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-03-11T22:26:08.589 INFO:teuthology.orchestra.run.smithi059.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2020-03-11T22:26:08.589 INFO:teuthology.orchestra.run.smithi059.stdout: 2020-03-11T22:26:08.590 INFO:teuthology.lock.ops:Updating smithi059.front.sepia.ceph.com on lock server 2020-03-11T22:26:08.608 INFO:teuthology.task.internal.lock_machines:1 smithi machines locked this try, 1/1 locked so far 2020-03-11T22:26:08.630 INFO:teuthology.task.internal.lock_machines:Locked targets: smithi059.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDq0KyHaYH4x9cuXUYcneBow7hlJNdkUOtkKUY26XA7bcdctYPZsthuXQEtLdfiKzDs91A8LfqXoaCENcOvLS7kr0QC2fz30mCKQ0bKDpMJ1AXM5gwJy/hpUZNrpTkRxV2yKiLLVaZCH7y2F12J4lEmNQY0Fl2GOUV+QxTx6IKPN2FkmgdLUpEZx1S6rrLdigTqBOCsAMA38UVfqkc4ZS9yAH8SpjAVtNIhUa02iVFFkWa9miCNy8CcQdpwRfFRfX55fhkqwu2iGcC9CUkZm+TFZfDZ1ywqu0n4TFwDSlkRJdecIryldzepZbCK2uOTm5oISLm8LStj2A8Ox1LnkV4c4rdZqCEd0u97meG590KbBOyrwfkueI13VJ+KZEUNiqrS+9iuXozqpRZxYB6Fc2QH2Kh8pOSpg+rRbDNGoyPSxUp7N9OwSGI1GeD56Q0NhcGeiRvwSnwtthhjuIJ8jP3q0T8IxubeJXGIJpF+kSI5R1+N1Ea3Cq5pUpH1FYRUf6M= 2020-03-11T22:26:08.630 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-03-11T22:26:08.686 INFO:teuthology.run_tasks:Running task internal.save_config... 2020-03-11T22:26:08.691 INFO:teuthology.task.internal:Saving configuration 2020-03-11T22:26:08.702 INFO:teuthology.run_tasks:Running task internal.check_lock... 2020-03-11T22:26:08.710 INFO:teuthology.task.internal.check_lock:Checking locks... 2020-03-11T22:26:08.726 DEBUG:teuthology.task.internal.check_lock:machine status is {u'is_vm': False, u'locked': True, u'locked_since': u'2020-03-11 22:21:27.125388', u'locked_by': u'scheduled_dzafman@bionic-laptop', u'up': True, u'mac_address': None, u'name': u'smithi059.front.sepia.ceph.com', u'os_version': u'8', u'machine_type': u'smithi', u'vm_host': None, u'os_type': u'centos', u'arch': u'x86_64', u'ssh_pub_key': u'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFVsCB1Ck8p/BwcAX3k8rcMvXmHC+GrxC8i4zzlHYeeTz4NvfExiU7UVZSA5j83NaWtxHiyupiNQCV8ZhonqxMQ=', u'description': u'/home/teuthworker/archive/dzafman-2020-03-11_15:13:33-rados-wip-zafman-octo-diag-distro-basic-smithi/4848256'} 2020-03-11T22:26:08.727 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2020-03-11T22:26:08.736 INFO:teuthology.task.internal:roles: ubuntu@smithi059.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2020-03-11T22:26:08.736 INFO:teuthology.run_tasks:Running task console_log... 2020-03-11T22:26:08.780 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=, signals=[15]) 2020-03-11T22:26:08.780 INFO:teuthology.run_tasks:Running task internal.connect... 2020-03-11T22:26:08.795 INFO:teuthology.task.internal:Opening connections... 2020-03-11T22:26:08.796 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi059.front.sepia.ceph.com 2020-03-11T22:26:08.797 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi059.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:26:08.871 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2020-03-11T22:26:08.882 INFO:teuthology.orchestra.run.smithi059:> uname -m 2020-03-11T22:26:09.049 INFO:teuthology.orchestra.run.smithi059.stdout:x86_64 2020-03-11T22:26:09.049 INFO:teuthology.orchestra.run.smithi059:> cat /etc/os-release 2020-03-11T22:26:09.104 INFO:teuthology.orchestra.run.smithi059.stdout:NAME="CentOS Linux" 2020-03-11T22:26:09.104 INFO:teuthology.orchestra.run.smithi059.stdout:VERSION="8 (Core)" 2020-03-11T22:26:09.104 INFO:teuthology.orchestra.run.smithi059.stdout:ID="centos" 2020-03-11T22:26:09.105 INFO:teuthology.orchestra.run.smithi059.stdout:ID_LIKE="rhel fedora" 2020-03-11T22:26:09.105 INFO:teuthology.orchestra.run.smithi059.stdout:VERSION_ID="8" 2020-03-11T22:26:09.105 INFO:teuthology.orchestra.run.smithi059.stdout:PLATFORM_ID="platform:el8" 2020-03-11T22:26:09.105 INFO:teuthology.orchestra.run.smithi059.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2020-03-11T22:26:09.106 INFO:teuthology.orchestra.run.smithi059.stdout:ANSI_COLOR="0;31" 2020-03-11T22:26:09.106 INFO:teuthology.orchestra.run.smithi059.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2020-03-11T22:26:09.106 INFO:teuthology.orchestra.run.smithi059.stdout:HOME_URL="https://www.centos.org/" 2020-03-11T22:26:09.106 INFO:teuthology.orchestra.run.smithi059.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-03-11T22:26:09.106 INFO:teuthology.orchestra.run.smithi059.stdout: 2020-03-11T22:26:09.107 INFO:teuthology.orchestra.run.smithi059.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2020-03-11T22:26:09.107 INFO:teuthology.orchestra.run.smithi059.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2020-03-11T22:26:09.107 INFO:teuthology.orchestra.run.smithi059.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-03-11T22:26:09.107 INFO:teuthology.orchestra.run.smithi059.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2020-03-11T22:26:09.108 INFO:teuthology.orchestra.run.smithi059.stdout: 2020-03-11T22:26:09.108 INFO:teuthology.lock.ops:Updating smithi059.front.sepia.ceph.com on lock server 2020-03-11T22:26:09.126 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2020-03-11T22:26:09.145 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2020-03-11T22:26:09.154 INFO:teuthology.task.internal:Checking for old test directory... 2020-03-11T22:26:09.155 INFO:teuthology.orchestra.run.smithi059:> test '!' -e /home/ubuntu/cephtest 2020-03-11T22:26:09.169 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2020-03-11T22:26:09.177 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2020-03-11T22:26:09.178 INFO:teuthology.orchestra.run.smithi059:> test -z $(ls -A /var/lib/ceph) 2020-03-11T22:26:09.238 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2020-03-11T22:26:09.264 INFO:teuthology.run_tasks:Running task kernel... 2020-03-11T22:26:09.291 DEBUG:teuthology.task.kernel:normalized config {'mon.a': {'sha1': 'distro', 'kdb': True}, 'osd.2': {'sha1': 'distro', 'kdb': True}, 'osd.1': {'sha1': 'distro', 'kdb': True}, 'osd.0': {'sha1': 'distro', 'kdb': True}, 'client.0': {'sha1': 'distro', 'kdb': True}, 'mgr.x': {'sha1': 'distro', 'kdb': True}} 2020-03-11T22:26:09.291 INFO:teuthology.task.kernel:config {'mon.a': {'sha1': 'distro', 'kdb': True}}, timeout 300 2020-03-11T22:26:09.291 INFO:teuthology.orchestra.run.smithi059:> uname -r 2020-03-11T22:26:09.306 INFO:teuthology.orchestra.run.smithi059.stdout:4.18.0-147.5.1.el8_1.x86_64 2020-03-11T22:26:09.307 INFO:teuthology.task.kernel:Running kernel on smithi059: 4.18.0-147.5.1.el8_1.x86_64 2020-03-11T22:26:09.307 INFO:teuthology.orchestra.run.smithi059:> sudo yum install -y kernel 2020-03-11T22:26:11.903 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 1 day, 7:00:18 ago on Tue 10 Mar 2020 03:25:53 PM UTC. 2020-03-11T22:26:12.285 INFO:teuthology.orchestra.run.smithi059.stdout:Package kernel-4.18.0-147.el8.x86_64 is already installed. 2020-03-11T22:26:12.285 INFO:teuthology.orchestra.run.smithi059.stdout:Package kernel-4.18.0-147.5.1.el8_1.x86_64 is already installed. 2020-03-11T22:26:12.446 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2020-03-11T22:26:12.448 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2020-03-11T22:26:12.448 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2020-03-11T22:26:12.589 INFO:teuthology.orchestra.run.smithi059:> echo no | sudo yum reinstall kernel || true 2020-03-11T22:26:13.006 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 1 day, 7:00:20 ago on Tue 10 Mar 2020 03:25:53 PM UTC. 2020-03-11T22:26:13.380 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2020-03-11T22:26:13.381 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2020-03-11T22:26:13.381 INFO:teuthology.orchestra.run.smithi059.stdout: Package Architecture Version Repository Size 2020-03-11T22:26:13.381 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2020-03-11T22:26:13.381 INFO:teuthology.orchestra.run.smithi059.stdout:Reinstalling: 2020-03-11T22:26:13.381 INFO:teuthology.orchestra.run.smithi059.stdout: kernel x86_64 4.18.0-147.5.1.el8_1 CentOS-Base 1.5 M 2020-03-11T22:26:13.381 INFO:teuthology.orchestra.run.smithi059.stdout: kernel x86_64 4.18.0-147.el8 CentOS-Base 1.5 M 2020-03-11T22:26:13.382 INFO:teuthology.orchestra.run.smithi059.stdout: 2020-03-11T22:26:13.382 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2020-03-11T22:26:13.382 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2020-03-11T22:26:13.382 INFO:teuthology.orchestra.run.smithi059.stdout: 2020-03-11T22:26:13.383 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 3.1 M 2020-03-11T22:26:13.383 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 0 2020-03-11T22:26:13.383 INFO:teuthology.orchestra.run.smithi059.stderr:Operation aborted. 2020-03-11T22:26:13.479 INFO:teuthology.orchestra.run.smithi059.stdout:Is this ok [y/N]: 2020-03-11T22:26:13.479 INFO:teuthology.orchestra.run.smithi059:> sudo yum reinstall -y kernel || true 2020-03-11T22:26:13.894 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 1 day, 7:00:20 ago on Tue 10 Mar 2020 03:25:53 PM UTC. 2020-03-11T22:26:14.303 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2020-03-11T22:26:14.304 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2020-03-11T22:26:14.304 INFO:teuthology.orchestra.run.smithi059.stdout: Package Architecture Version Repository Size 2020-03-11T22:26:14.304 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2020-03-11T22:26:14.304 INFO:teuthology.orchestra.run.smithi059.stdout:Reinstalling: 2020-03-11T22:26:14.304 INFO:teuthology.orchestra.run.smithi059.stdout: kernel x86_64 4.18.0-147.5.1.el8_1 CentOS-Base 1.5 M 2020-03-11T22:26:14.305 INFO:teuthology.orchestra.run.smithi059.stdout: kernel x86_64 4.18.0-147.el8 CentOS-Base 1.5 M 2020-03-11T22:26:14.305 INFO:teuthology.orchestra.run.smithi059.stdout: 2020-03-11T22:26:14.305 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2020-03-11T22:26:14.305 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2020-03-11T22:26:14.305 INFO:teuthology.orchestra.run.smithi059.stdout: 2020-03-11T22:26:14.306 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 3.1 M 2020-03-11T22:26:14.306 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 0 2020-03-11T22:26:14.306 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2020-03-11T22:26:14.382 INFO:teuthology.orchestra.run.smithi059.stdout:(1/2): kernel-4.18.0-147.5.1.el8_1.x86_64.rpm 22 MB/s | 1.5 MB 00:00 2020-03-11T22:26:14.392 INFO:teuthology.orchestra.run.smithi059.stdout:(2/2): kernel-4.18.0-147.el8.x86_64.rpm 19 MB/s | 1.5 MB 00:00 2020-03-11T22:26:14.392 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:26:14.393 INFO:teuthology.orchestra.run.smithi059.stdout:Total 35 MB/s | 3.1 MB 00:00 2020-03-11T22:26:14.427 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2020-03-11T22:26:14.624 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2020-03-11T22:26:14.624 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2020-03-11T22:26:14.692 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2020-03-11T22:26:14.692 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2020-03-11T22:26:14.863 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2020-03-11T22:26:15.033 INFO:teuthology.orchestra.run.smithi059.stdout: Reinstalling : kernel-4.18.0-147.el8.x86_64 1/4 2020-03-11T22:26:15.141 INFO:teuthology.orchestra.run.smithi059.stdout: Reinstalling : kernel-4.18.0-147.5.1.el8_1.x86_64 2/4 2020-03-11T22:26:15.142 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : kernel-4.18.0-147.el8.x86_64 3/4 2020-03-11T22:26:15.276 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: kernel-4.18.0-147.el8.x86_64 3/4 2020-03-11T22:26:15.276 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : kernel-4.18.0-147.5.1.el8_1.x86_64 4/4 2020-03-11T22:26:15.514 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: kernel-4.18.0-147.5.1.el8_1.x86_64 4/4 2020-03-11T22:26:15.515 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : kernel-4.18.0-147.5.1.el8_1.x86_64 1/4 2020-03-11T22:26:15.515 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : kernel-4.18.0-147.5.1.el8_1.x86_64 2/4 2020-03-11T22:26:15.515 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : kernel-4.18.0-147.el8.x86_64 3/4 2020-03-11T22:26:15.764 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : kernel-4.18.0-147.el8.x86_64 4/4 2020-03-11T22:26:15.764 INFO:teuthology.orchestra.run.smithi059.stdout: 2020-03-11T22:26:15.764 INFO:teuthology.orchestra.run.smithi059.stdout:Reinstalled: 2020-03-11T22:26:15.765 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-4.18.0-147.5.1.el8_1.x86_64 kernel-4.18.0-147.el8.x86_64 2020-03-11T22:26:15.765 INFO:teuthology.orchestra.run.smithi059.stdout: 2020-03-11T22:26:15.765 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2020-03-11T22:26:15.870 INFO:teuthology.orchestra.run.smithi059:> rpm -q kernel --last | head -n 1 2020-03-11T22:26:15.923 INFO:teuthology.orchestra.run.smithi059.stdout:kernel-4.18.0-147.5.1.el8_1.x86_64 Wed 11 Mar 2020 10:26:15 PM UTC 2020-03-11T22:26:15.924 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-147.5.1.el8_1.x86_64 2020-03-11T22:26:15.925 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2020-03-11T22:26:15.925 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2020-03-11T22:26:15.925 INFO:teuthology.orchestra.run.smithi059:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2020-03-11T22:26:15.963 INFO:teuthology.orchestra.run.smithi059.stdout:ttyS1 2020-03-11T22:26:15.980 INFO:teuthology.run_tasks:Running task internal.base... 2020-03-11T22:26:15.992 INFO:teuthology.task.internal:Creating test directory... 2020-03-11T22:26:15.992 INFO:teuthology.orchestra.run.smithi059:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2020-03-11T22:26:16.035 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2020-03-11T22:26:16.044 INFO:teuthology.run_tasks:Running task internal.archive... 2020-03-11T22:26:16.056 INFO:teuthology.task.internal:Creating archive directory... 2020-03-11T22:26:16.057 INFO:teuthology.orchestra.run.smithi059:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2020-03-11T22:26:16.095 INFO:teuthology.run_tasks:Running task internal.coredump... 2020-03-11T22:26:16.104 INFO:teuthology.task.internal:Enabling coredump saving... 2020-03-11T22:26:16.105 INFO:teuthology.orchestra.run.smithi059:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2020-03-11T22:26:16.216 INFO:teuthology.orchestra.run.smithi059.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:26:16.233 INFO:teuthology.orchestra.run.smithi059.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:26:16.234 INFO:teuthology.run_tasks:Running task internal.sudo... 2020-03-11T22:26:16.243 INFO:teuthology.task.internal:Configuring sudo... 2020-03-11T22:26:16.243 INFO:teuthology.orchestra.run.smithi059:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2020-03-11T22:26:16.274 INFO:teuthology.run_tasks:Running task internal.syslog... 2020-03-11T22:26:16.286 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2020-03-11T22:26:16.286 INFO:teuthology.orchestra.run.smithi059:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2020-03-11T22:26:16.329 INFO:teuthology.orchestra.run.smithi059:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2020-03-11T22:26:16.403 INFO:teuthology.orchestra.run.smithi059:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2020-03-11T22:26:16.468 INFO:teuthology.orchestra.run.smithi059:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2020-03-11T22:26:16.541 INFO:teuthology.orchestra.run.smithi059:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2020-03-11T22:26:16.595 INFO:teuthology.orchestra.run.smithi059:> sudo sh -c 'cat > /etc/rsyslog.d/80-cephtest.conf' 2020-03-11T22:26:16.680 INFO:teuthology.orchestra.run.smithi059:> sudo service rsyslog restart 2020-03-11T22:26:16.784 INFO:teuthology.orchestra.run.smithi059.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2020-03-11T22:26:17.089 INFO:teuthology.run_tasks:Running task internal.timer... 2020-03-11T22:26:17.097 INFO:teuthology.task.internal:Starting timer... 2020-03-11T22:26:17.097 INFO:teuthology.run_tasks:Running task pcp... 2020-03-11T22:26:17.128 INFO:teuthology.run_tasks:Running task selinux... 2020-03-11T22:26:17.139 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2020-03-11T22:26:17.139 INFO:teuthology.orchestra.run.smithi059:> sudo service auditd rotate 2020-03-11T22:26:17.225 INFO:teuthology.orchestra.run.smithi059.stdout:Rotating logs: [ OK ] 2020-03-11T22:26:17.227 DEBUG:teuthology.task.selinux:Getting current SELinux state 2020-03-11T22:26:17.227 INFO:teuthology.orchestra.run.smithi059:> /usr/sbin/getenforce 2020-03-11T22:26:17.244 INFO:teuthology.orchestra.run.smithi059.stdout:Permissive 2020-03-11T22:26:17.245 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi059.front.sepia.ceph.com': 'permissive'} 2020-03-11T22:26:17.245 INFO:teuthology.orchestra.run.smithi059:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|scontext=system_u:system_r:logrotate_t:s0\)' 2020-03-11T22:26:17.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-03-11T22:26:17.317 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2020-03-11T22:26:17.317 INFO:teuthology.orchestra.run.smithi059:> sudo /usr/sbin/setenforce permissive 2020-03-11T22:26:17.389 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2020-03-11T22:26:17.399 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated; assuming it is current 2020-03-11T22:26:17.399 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to branch origin/master 2020-03-11T22:26:17.423 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}] 2020-03-11T22:26:17.429 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi059.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2020-03-11T22:26:17.798 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2020-03-11T22:26:18.993 INFO:teuthology.task.ansible.out: [WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2020-03-11T22:26:21.961 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2020-03-11T22:26:21.983 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:21 +0000 (0:00:00.064) 0:00:00.064 ******* 2020-03-11T22:26:24.580 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi059.front.sepia.ceph.com] 2020-03-11T22:26:24.671 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:24 +0000 (0:00:02.688) 0:00:02.753 ******* 2020-03-11T22:26:25.051 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2020-03-11T22:26:25.053 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2020-03-11T22:26:25.088 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:25 +0000 (0:00:00.416) 0:00:03.169 ******* 2020-03-11T22:26:25.712 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** ok: [smithi059.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2020-03-11T22:26:25.748 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:25 +0000 (0:00:00.660) 0:00:03.830 ******* 2020-03-11T22:26:25.774 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2020-03-11T22:26:25.776 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:25.811 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:25 +0000 (0:00:00.062) 0:00:03.892 ******* 2020-03-11T22:26:26.157 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** 2020-03-11T22:26:26.157 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-03-11T22:26:26.196 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:26 +0000 (0:00:00.384) 0:00:04.277 ******* 2020-03-11T22:26:27.252 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2020-03-11T22:26:27.253 INFO:teuthology.task.ansible.out:changed: [smithi059.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1583965586.21-44358518816120/source", "state": "file", "uid": 0} 2020-03-11T22:26:27.335 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:27 +0000 (0:00:01.139) 0:00:05.416 ******* 2020-03-11T22:26:27.781 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2020-03-11T22:26:27.786 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2020-03-11T22:26:27.788 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2020-03-11T22:26:27.895 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2020-03-11T22:26:27.901 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2020-03-11T22:26:28.085 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:28 +0000 (0:00:00.750) 0:00:06.167 ******* 2020-03-11T22:26:28.762 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi059.front.sepia.ceph.com] 2020-03-11T22:26:28.844 INFO:teuthology.task.ansible.out: [WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. 2020-03-11T22:26:28.846 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:28 +0000 (0:00:00.759) 0:00:06.926 ******* 2020-03-11T22:26:29.363 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2020-03-11T22:26:29.363 INFO:teuthology.task.ansible.out:changed: [smithi059.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.223790", "end": "2020-03-11 22:26:29.341261", "rc": 0, "start": "2020-03-11 22:26:29.117471", "stderr": "", "stderr_lines": [], "stdout": "d6cc316c8d11cadac2a03d4f84b215a35cf6c4b6", "stdout_lines": ["d6cc316c8d11cadac2a03d4f84b215a35cf6c4b6"]} 2020-03-11T22:26:29.400 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:29 +0000 (0:00:00.555) 0:00:07.481 ******* 2020-03-11T22:26:29.631 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2020-03-11T22:26:29.633 INFO:teuthology.task.ansible.out:changed: [smithi059.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.012079", "end": "2020-03-11 22:26:29.601750", "failed_when_result": false, "rc": 0, "start": "2020-03-11 22:26:29.589671", "stderr": "", "stderr_lines": [], "stdout": "d6cc316c8d11cadac2a03d4f84b215a35cf6c4b6", "stdout_lines": ["d6cc316c8d11cadac2a03d4f84b215a35cf6c4b6"]} 2020-03-11T22:26:29.668 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:29 +0000 (0:00:00.268) 0:00:07.750 ******* 2020-03-11T22:26:29.704 INFO:teuthology.task.ansible.out: TASK [users : Determine if we can skip users and pubkeys updates] ************** 2020-03-11T22:26:29.706 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => {"ansible_facts": {"perform_users_role": false}, "changed": false} 2020-03-11T22:26:29.741 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:29 +0000 (0:00:00.072) 0:00:07.822 ******* 2020-03-11T22:26:29.766 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** 2020-03-11T22:26:29.768 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:29.804 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:29 +0000 (0:00:00.063) 0:00:07.886 ******* 2020-03-11T22:26:29.830 INFO:teuthology.task.ansible.out: TASK [users : Remove managed_admin_users from managed_users] ******************* 2020-03-11T22:26:29.832 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:29.866 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:29 +0000 (0:00:00.061) 0:00:07.948 ******* 2020-03-11T22:26:29.892 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_users list] *********************************** 2020-03-11T22:26:29.893 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:29.929 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:29 +0000 (0:00:00.062) 0:00:08.011 ******* 2020-03-11T22:26:29.956 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_admin_users list] ***************************** 2020-03-11T22:26:29.957 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:29.993 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:29 +0000 (0:00:00.064) 0:00:08.075 ******* 2020-03-11T22:26:30.019 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2020-03-11T22:26:30.020 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.059 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:30 +0000 (0:00:00.066) 0:00:08.141 ******* 2020-03-11T22:26:30.116 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2020-03-11T22:26:30.117 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.118 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.130 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.141 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.141 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.152 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.153 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.164 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.175 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.176 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.187 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.197 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.198 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.209 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.220 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.220 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.232 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.232 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.244 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.254 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.265 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.266 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.277 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.278 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.289 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.289 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.301 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.311 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.312 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.323 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.334 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.335 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.346 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.357 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.358 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.369 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.382 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.392 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.403 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.404 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.415 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.426 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.428 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.438 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.439 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.450 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.460 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.461 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.472 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.472 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.484 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.495 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.495 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.506 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.517 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.518 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.520 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.530 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.531 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.541 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.542 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.558 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.566 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.577 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.577 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.588 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.589 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.600 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.611 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.611 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.613 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.623 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.624 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.635 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.636 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.647 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.659 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.660 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.671 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.672 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.683 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.684 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.696 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.696 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.708 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.709 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.720 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.721 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.722 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.733 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.744 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.745 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.756 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.757 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.761 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.769 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.780 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.780 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.791 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.792 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42', u'name': u'zdover'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.802 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b', u'name': u'rraja'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.803 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f', u'name': u'toabctl'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.814 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39', u'name': u'mgfritch'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.814 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d', u'name': u'xiubli'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.825 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f', u'name': u'jafaj'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.826 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef', u'name': u'jschmid'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.837 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'kachau@thinkpad GsZIGCdLojpvOZDxwcWSlg 0638220db65953e0cd1f134c499e4a713b32a882c8a92458afcb495dbd087518', u'name': u'kachau'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad GsZIGCdLojpvOZDxwcWSlg 0638220db65953e0cd1f134c499e4a713b32a882c8a92458afcb495dbd087518"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.837 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034', u'name': u'khiremat'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.848 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'ovpn': u'kapandya@thinkpad elavnUlTLKzuE7/eJ2nxMw 4ac31f4c197b2b6325a404950175faeedc105ea45092672e924b6edafe90af23', u'name': u'kapandya'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad elavnUlTLKzuE7/eJ2nxMw 4ac31f4c197b2b6325a404950175faeedc105ea45092672e924b6edafe90af23"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:30.903 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:30 +0000 (0:00:00.844) 0:00:08.985 ******* 2020-03-11T22:26:30.929 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2020-03-11T22:26:30.968 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:30 +0000 (0:00:00.064) 0:00:09.050 ******* 2020-03-11T22:26:30.995 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* 2020-03-11T22:26:30.996 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:31.035 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:31 +0000 (0:00:00.066) 0:00:09.117 ******* 2020-03-11T22:26:31.061 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2020-03-11T22:26:31.063 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:31.100 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:31 +0000 (0:00:00.064) 0:00:09.182 ******* 2020-03-11T22:26:31.126 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-03-11T22:26:31.127 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:31.163 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:31 +0000 (0:00:00.063) 0:00:09.245 ******* 2020-03-11T22:26:31.199 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-03-11T22:26:31.201 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:31.237 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:31 +0000 (0:00:00.073) 0:00:09.319 ******* 2020-03-11T22:26:31.263 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** 2020-03-11T22:26:31.264 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:31.307 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:31 +0000 (0:00:00.070) 0:00:09.389 ******* 2020-03-11T22:26:31.344 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-03-11T22:26:31.345 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "skip_reason": "Conditional result was False"} skipping: [smithi059.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "skip_reason": "Conditional result was False"} skipping: [smithi059.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "skip_reason": "Conditional result was False"} 2020-03-11T22:26:31.356 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "skip_reason": "Conditional result was False"} 2020-03-11T22:26:31.357 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "skip_reason": "Conditional result was False"} 2020-03-11T22:26:31.369 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "skip_reason": "Conditional result was False"} 2020-03-11T22:26:31.381 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "skip_reason": "Conditional result was False"} 2020-03-11T22:26:31.382 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "skip_reason": "Conditional result was False"} 2020-03-11T22:26:31.393 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "skip_reason": "Conditional result was False"} 2020-03-11T22:26:31.394 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "skip_reason": "Conditional result was False"} 2020-03-11T22:26:31.406 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "skip_reason": "Conditional result was False"} 2020-03-11T22:26:31.406 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "skip_reason": "Conditional result was False"} 2020-03-11T22:26:31.417 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "skip_reason": "Conditional result was False"} 2020-03-11T22:26:31.418 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "skip_reason": "Conditional result was False"} 2020-03-11T22:26:31.428 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "skip_reason": "Conditional result was False"} 2020-03-11T22:26:31.429 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "skip_reason": "Conditional result was False"} 2020-03-11T22:26:31.440 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "skip_reason": "Conditional result was False"} 2020-03-11T22:26:31.441 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "skip_reason": "Conditional result was False"} 2020-03-11T22:26:31.451 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "skip_reason": "Conditional result was False"} 2020-03-11T22:26:31.452 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "skip_reason": "Conditional result was False"} 2020-03-11T22:26:31.453 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "skip_reason": "Conditional result was False"} 2020-03-11T22:26:31.463 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "skip_reason": "Conditional result was False"} 2020-03-11T22:26:31.464 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "skip_reason": "Conditional result was False"} 2020-03-11T22:26:31.475 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "skip_reason": "Conditional result was False"} 2020-03-11T22:26:31.475 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "skip_reason": "Conditional result was False"} 2020-03-11T22:26:31.486 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "skip_reason": "Conditional result was False"} 2020-03-11T22:26:31.487 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "skip_reason": "Conditional result was False"} 2020-03-11T22:26:31.498 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "skip_reason": "Conditional result was False"} 2020-03-11T22:26:31.508 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "skip_reason": "Conditional result was False"} 2020-03-11T22:26:31.509 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "skip_reason": "Conditional result was False"} 2020-03-11T22:26:31.520 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "skip_reason": "Conditional result was False"} 2020-03-11T22:26:31.521 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "skip_reason": "Conditional result was False"} 2020-03-11T22:26:31.531 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "skip_reason": "Conditional result was False"} 2020-03-11T22:26:31.531 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "skip_reason": "Conditional result was False"} 2020-03-11T22:26:31.542 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "skip_reason": "Conditional result was False"} 2020-03-11T22:26:31.543 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "skip_reason": "Conditional result was False"} 2020-03-11T22:26:31.562 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "skip_reason": "Conditional result was False"} skipping: [smithi059.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "skip_reason": "Conditional result was False"} 2020-03-11T22:26:31.567 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "skip_reason": "Conditional result was False"} 2020-03-11T22:26:31.568 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "skip_reason": "Conditional result was False"} 2020-03-11T22:26:31.569 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "skip_reason": "Conditional result was False"} 2020-03-11T22:26:31.579 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "skip_reason": "Conditional result was False"} 2020-03-11T22:26:31.624 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:31 +0000 (0:00:00.316) 0:00:09.705 ******* 2020-03-11T22:26:32.474 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2020-03-11T22:26:32.475 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => {"changed": false, "checksum": "9c97a4e7e2fa12022ca70c847609415d5b1cb1c9", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/keys-repo-sha1", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "state": "file", "uid": 0} 2020-03-11T22:26:32.514 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:32 +0000 (0:00:00.890) 0:00:10.595 ******* 2020-03-11T22:26:32.542 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2020-03-11T22:26:32.544 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => { "msg": "Host smithi059.front.sepia.ceph.com is running CentOS 8.1 (Core)" } 2020-03-11T22:26:32.584 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:32 +0000 (0:00:00.070) 0:00:10.666 ******* 2020-03-11T22:26:32.624 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2020-03-11T22:26:32.625 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml) => {"ansible_facts": {"nrpe_selinux_packages": ["python3-libsemanage", "python3-policycoreutils"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"} 2020-03-11T22:26:32.672 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:32 +0000 (0:00:00.087) 0:00:10.753 ******* 2020-03-11T22:26:32.698 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2020-03-11T22:26:32.700 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:32.743 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:32 +0000 (0:00:00.070) 0:00:10.824 ******* 2020-03-11T22:26:33.040 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2020-03-11T22:26:33.040 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.082572", "end": "2020-03-11 22:26:33.010782", "rc": 0, "start": "2020-03-11 22:26:32.928210", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2020-03-11T22:26:33.095 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:33 +0000 (0:00:00.352) 0:00:11.177 ******* 2020-03-11T22:26:33.122 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2020-03-11T22:26:33.124 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:33.161 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:33 +0000 (0:00:00.065) 0:00:11.243 ******* 2020-03-11T22:26:33.208 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2020-03-11T22:26:33.210 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:33.248 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:33 +0000 (0:00:00.086) 0:00:11.329 ******* 2020-03-11T22:26:33.275 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2020-03-11T22:26:33.276 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:33.331 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:33 +0000 (0:00:00.083) 0:00:11.413 ******* 2020-03-11T22:26:33.367 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2020-03-11T22:26:33.369 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:33.404 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:33 +0000 (0:00:00.072) 0:00:11.485 ******* 2020-03-11T22:26:33.440 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2020-03-11T22:26:33.441 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:33.478 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:33 +0000 (0:00:00.074) 0:00:11.559 ******* 2020-03-11T22:26:33.504 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2020-03-11T22:26:33.505 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:33.541 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:33 +0000 (0:00:00.063) 0:00:11.623 ******* 2020-03-11T22:26:33.577 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2020-03-11T22:26:33.579 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi059.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-11T22:26:33.615 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:33 +0000 (0:00:00.073) 0:00:11.697 ******* 2020-03-11T22:26:33.651 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:33.688 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:33 +0000 (0:00:00.073) 0:00:11.770 ******* 2020-03-11T22:26:33.724 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2020-03-11T22:26:33.726 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:33.762 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:33 +0000 (0:00:00.073) 0:00:11.843 ******* 2020-03-11T22:26:33.787 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2020-03-11T22:26:33.789 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-11T22:26:33.826 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:33 +0000 (0:00:00.064) 0:00:11.907 ******* 2020-03-11T22:26:33.863 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2020-03-11T22:26:33.865 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:33.910 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:33 +0000 (0:00:00.084) 0:00:11.991 ******* 2020-03-11T22:26:33.936 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2020-03-11T22:26:33.938 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-11T22:26:33.973 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:33 +0000 (0:00:00.062) 0:00:12.054 ******* 2020-03-11T22:26:33.998 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2020-03-11T22:26:34.000 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:34.037 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:34 +0000 (0:00:00.064) 0:00:12.119 ******* 2020-03-11T22:26:34.066 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2020-03-11T22:26:34.068 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:34.102 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:34 +0000 (0:00:00.065) 0:00:12.184 ******* 2020-03-11T22:26:34.129 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2020-03-11T22:26:34.130 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:34.166 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:34 +0000 (0:00:00.063) 0:00:12.248 ******* 2020-03-11T22:26:34.192 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2020-03-11T22:26:34.194 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:34.230 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:34 +0000 (0:00:00.063) 0:00:12.311 ******* 2020-03-11T22:26:34.231 INFO:teuthology.task.ansible.out: 2020-03-11T22:26:34.256 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2020-03-11T22:26:34.257 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-11T22:26:34.293 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:34 +0000 (0:00:00.062) 0:00:12.374 ******* 2020-03-11T22:26:34.319 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-03-11T22:26:34.320 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:34.356 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:34 +0000 (0:00:00.062) 0:00:12.437 ******* 2020-03-11T22:26:34.383 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2020-03-11T22:26:34.384 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:34.421 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:34 +0000 (0:00:00.064) 0:00:12.502 ******* 2020-03-11T22:26:34.449 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:34.483 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:34 +0000 (0:00:00.062) 0:00:12.565 ******* 2020-03-11T22:26:34.519 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2020-03-11T22:26:34.521 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:34.561 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:34 +0000 (0:00:00.077) 0:00:12.643 ******* 2020-03-11T22:26:34.587 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2020-03-11T22:26:34.589 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:34.626 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:34 +0000 (0:00:00.064) 0:00:12.707 ******* 2020-03-11T22:26:34.662 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2020-03-11T22:26:34.664 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:34.702 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:34 +0000 (0:00:00.075) 0:00:12.783 ******* 2020-03-11T22:26:34.728 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2020-03-11T22:26:34.729 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-11T22:26:34.777 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:34 +0000 (0:00:00.071) 0:00:12.855 ******* 2020-03-11T22:26:34.800 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2020-03-11T22:26:34.845 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:34 +0000 (0:00:00.072) 0:00:12.927 ******* 2020-03-11T22:26:34.882 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2020-03-11T22:26:34.884 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:34.944 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:34 +0000 (0:00:00.098) 0:00:13.025 ******* 2020-03-11T22:26:35.171 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2020-03-11T22:26:35.171 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-03-11T22:26:35.213 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:35 +0000 (0:00:00.268) 0:00:13.294 ******* 2020-03-11T22:26:35.762 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2020-03-11T22:26:35.763 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-mirrorlist', u'name': u'Extra Packages for Enterprise Linux', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': u'1', u'baseurl': u'http://dl.fedoraproject.org/pub/epel/8/Everything/$basearch'}, 'key': u'epel'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "2da7f101e29cfe8f3b630dbd6da82b4f9a749fee", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/8/Everything/$basearch", "enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 304, "state": "file", "uid": 0} 2020-03-11T22:26:36.255 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-testing-mirrorlist', u'name': u'Extra Packages for Enterprise Linux - Testing', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': 0, u'baseurl': u'http://dl.fedoraproject.org/pub/epel/testing/8/Everything/$basearch'}, 'key': u'epel-testing'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "ddfaa38799b2dd4c23933a375aa4c3ff61bf7625", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/testing/8/Everything/$basearch", "enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 338, "state": "file", "uid": 0} 2020-03-11T22:26:36.297 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:36 +0000 (0:00:01.084) 0:00:14.378 ******* 2020-03-11T22:26:36.830 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2020-03-11T22:26:36.832 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "3c9ba7bacd1e30bed014cefd1fb4eab0f0f8fa10", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 484, "state": "file", "uid": 0} 2020-03-11T22:26:37.336 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "d542c3e2c30a0d3eb7d454775342c302c0ded15b", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 540, "state": "file", "uid": 0} 2020-03-11T22:26:37.376 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:37 +0000 (0:00:01.079) 0:00:15.458 ******* 2020-03-11T22:26:37.403 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-03-11T22:26:37.404 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:37.440 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:37 +0000 (0:00:00.063) 0:00:15.521 ******* 2020-03-11T22:26:37.466 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-03-11T22:26:37.467 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:37.504 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:37 +0000 (0:00:00.064) 0:00:15.586 ******* 2020-03-11T22:26:37.530 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2020-03-11T22:26:37.533 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:37.568 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:37 +0000 (0:00:00.064) 0:00:15.650 ******* 2020-03-11T22:26:37.594 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2020-03-11T22:26:37.596 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:37.633 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:37 +0000 (0:00:00.064) 0:00:15.715 ******* 2020-03-11T22:26:37.662 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-03-11T22:26:37.663 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:37.703 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:37 +0000 (0:00:00.069) 0:00:15.784 ******* 2020-03-11T22:26:37.729 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2020-03-11T22:26:37.731 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:37.772 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:37 +0000 (0:00:00.069) 0:00:15.853 ******* 2020-03-11T22:26:37.798 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2020-03-11T22:26:37.799 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:37.837 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:37 +0000 (0:00:00.065) 0:00:15.919 ******* 2020-03-11T22:26:37.875 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2020-03-11T22:26:37.877 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:37.912 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:37 +0000 (0:00:00.075) 0:00:15.994 ******* 2020-03-11T22:26:37.938 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2020-03-11T22:26:37.939 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:37.975 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:37 +0000 (0:00:00.062) 0:00:16.057 ******* 2020-03-11T22:26:38.527 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2020-03-11T22:26:38.529 INFO:teuthology.task.ansible.out:changed: [smithi059.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.346155", "end": "2020-03-11 22:26:38.500312", "rc": 0, "start": "2020-03-11 22:26:38.154157", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:26:38.564 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:38 +0000 (0:00:00.588) 0:00:16.645 ******* 2020-03-11T22:26:39.864 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2020-03-11T22:26:39.866 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: krb5-workstation"]} 2020-03-11T22:26:39.902 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:39 +0000 (0:00:01.338) 0:00:17.983 ******* 2020-03-11T22:26:39.927 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2020-03-11T22:26:39.929 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:39.964 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:39 +0000 (0:00:00.062) 0:00:18.046 ******* 2020-03-11T22:26:39.991 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:40.029 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:40 +0000 (0:00:00.064) 0:00:18.110 ******* 2020-03-11T22:26:40.054 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2020-03-11T22:26:40.055 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:40.091 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:40 +0000 (0:00:00.062) 0:00:18.172 ******* 2020-03-11T22:26:40.116 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2020-03-11T22:26:40.118 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:40.158 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:40 +0000 (0:00:00.067) 0:00:18.240 ******* 2020-03-11T22:26:40.647 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2020-03-11T22:26:40.648 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2020-03-11T22:26:40.687 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:40 +0000 (0:00:00.528) 0:00:18.768 ******* 2020-03-11T22:26:41.672 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2020-03-11T22:26:41.674 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: smartmontools"]} 2020-03-11T22:26:41.711 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:41 +0000 (0:00:01.023) 0:00:19.792 ******* 2020-03-11T22:26:42.230 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2020-03-11T22:26:42.231 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2020-03-11T22:26:42.718 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2020-03-11T22:26:43.213 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2020-03-11T22:26:43.253 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:43 +0000 (0:00:01.541) 0:00:21.334 ******* 2020-03-11T22:26:43.464 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2020-03-11T22:26:43.465 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2020-03-11T22:26:43.504 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:43 +0000 (0:00:00.251) 0:00:21.586 ******* 2020-03-11T22:26:43.992 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2020-03-11T22:26:43.993 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "9a447e184389aadb9aaf1e77f30f0a7f8b716c4e", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "secontext": "system_u:object_r:bin_t:s0", "size": 12725, "state": "file", "uid": 0} 2020-03-11T22:26:44.471 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2020-03-11T22:26:44.930 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 3605, "state": "file", "uid": 0} 2020-03-11T22:26:44.932 INFO:teuthology.task.ansible.out: 2020-03-11T22:26:44.974 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:44 +0000 (0:00:01.469) 0:00:23.056 ******* 2020-03-11T22:26:45.001 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2020-03-11T22:26:45.002 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:45.045 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:45 +0000 (0:00:00.071) 0:00:23.127 ******* 2020-03-11T22:26:45.082 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2020-03-11T22:26:45.082 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/yum_systems.yml"], "changed": false} 2020-03-11T22:26:45.121 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:45 +0000 (0:00:00.075) 0:00:23.202 ******* 2020-03-11T22:26:45.332 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2020-03-11T22:26:45.334 INFO:teuthology.task.ansible.out:changed: [smithi059.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2020-03-11T22:26:45.372 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:45 +0000 (0:00:00.251) 0:00:23.454 ******* 2020-03-11T22:26:45.410 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2020-03-11T22:26:45.412 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-03-11T22:26:45.422 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-03-11T22:26:45.443 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-03-11T22:26:45.454 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-03-11T22:26:45.493 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:45 +0000 (0:00:00.120) 0:00:23.574 ******* 2020-03-11T22:26:46.470 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2020-03-11T22:26:46.472 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: nagios-common", "Installed: nrpe", "Installed: nagios-plugins", "Installed: nagios-plugins-load"]} 2020-03-11T22:26:46.508 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:46 +0000 (0:00:01.014) 0:00:24.589 ******* 2020-03-11T22:26:46.544 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2020-03-11T22:26:46.546 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:46.581 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:46 +0000 (0:00:00.073) 0:00:24.663 ******* 2020-03-11T22:26:47.074 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2020-03-11T22:26:47.075 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2020-03-11T22:26:47.114 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:47 +0000 (0:00:00.532) 0:00:25.195 ******* 2020-03-11T22:26:47.747 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2020-03-11T22:26:47.748 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2020-03-11T22:26:47.786 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:47 +0000 (0:00:00.672) 0:00:25.868 ******* 2020-03-11T22:26:47.822 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2020-03-11T22:26:47.824 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:47.862 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:47 +0000 (0:00:00.076) 0:00:25.944 ******* 2020-03-11T22:26:48.080 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2020-03-11T22:26:48.081 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-03-11T22:26:48.117 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:48 +0000 (0:00:00.254) 0:00:26.198 ******* 2020-03-11T22:26:48.370 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2020-03-11T22:26:48.372 INFO:teuthology.task.ansible.out:fatal: [smithi059.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2020-03-11T22:26:48.418 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:48 +0000 (0:00:00.301) 0:00:26.499 ******* 2020-03-11T22:26:48.447 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2020-03-11T22:26:48.448 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:26:48.486 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:48 +0000 (0:00:00.067) 0:00:26.567 ******* 2020-03-11T22:26:48.989 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2020-03-11T22:26:48.990 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => {"changed": false, "checksum": "333141ba5e85aea94f51511a52d9b115d2e0112f", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1174, "state": "file", "uid": 0} 2020-03-11T22:26:49.033 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:49 +0000 (0:00:00.547) 0:00:27.114 ******* 2020-03-11T22:26:49.800 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2020-03-11T22:26:49.802 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Wed 2020-03-11 22:25:37 UTC", "ActiveEnterTimestampMonotonic": "17343899", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-journald.socket network.target sysinit.target basic.target var-run.mount system.slice systemd-tmpfiles-setup.service nss-lookup.target tmp.mount local-fs.target -.mount time-sync.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Wed 2020-03-11 22:25:37 UTC", "AssertTimestampMonotonic": "17342843", "Before": "xdm.service multi-user.target getty@tty1.service shutdown.target plymouth-quit.service", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Wed 2020-03-11 22:25:37 UTC", "ConditionTimestampMonotonic": "17342843", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "D 2020-03-11T22:26:49.804 INFO:teuthology.task.ansible.out:efaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "2417", "ExecMainStartTimestamp": "Wed 2020-03-11 22:25:37 UTC", "ExecMainStartTimestampMonotonic": "17343832", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f ; ignore_errors=no ; start_time=[Wed 2020-03-11 22:25:37 UTC] ; stop_time=[n/a] ; pid=2417 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "GID": "986", "Group": "nrpe", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Wed 2020-03-11 22:25:37 UTC", "InactiveExitTimestampMonotonic": "17343899", "InvocationID": "7b29ab4fe5824a21827c3a39c277b762", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "Limi 2020-03-11T22:26:49.805 INFO:teuthology.task.ansible.out:tDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "127467", "LimitNPROCSoft": "127467", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127467", "LimitSIGPENDINGSoft": "127467", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "2417", "MemoryAccounting": "yes", "MemoryCurrent": "1110016", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nrpe/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount sysinit.target system.slice", "RequiresMountsFor": "/var/tmp /run/nrpe", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectory": "n 2020-03-11T22:26:49.807 INFO:teuthology.task.ansible.out:rpe", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Wed 2020-03-11 22:25:37 UTC", "StateChangeTimestampMonotonic": "17343899", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "26213", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "990", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Wed 2020-03-11 22:25:37 UTC", "WatchdogTimestampMonotonic": "17343898", "WatchdogUSec": "0"}} 2020-03-11T22:26:49.841 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:49 +0000 (0:00:00.808) 0:00:27.923 ******* 2020-03-11T22:26:50.049 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2020-03-11T22:26:50.051 INFO:teuthology.task.ansible.out:changed: [smithi059.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.001766", "end": "2020-03-11 22:26:50.024126", "rc": 0, "start": "2020-03-11 22:26:50.022360", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2020-03-11T22:26:50.087 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:50 +0000 (0:00:00.245) 0:00:28.168 ******* 2020-03-11T22:26:51.076 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** ok: [smithi059.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: python3-libsemanage", "Installed: python3-policycoreutils"]} 2020-03-11T22:26:51.112 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:51 +0000 (0:00:01.024) 0:00:29.193 ******* 2020-03-11T22:26:52.085 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2020-03-11T22:26:52.086 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: selinux-policy-targeted"]} 2020-03-11T22:26:52.123 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:52 +0000 (0:00:01.011) 0:00:30.204 ******* 2020-03-11T22:26:53.633 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2020-03-11T22:26:53.635 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2020-03-11T22:26:53.682 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:26:53 +0000 (0:00:01.554) 0:00:31.759 ******* 2020-03-11T22:27:10.665 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2020-03-11T22:27:10.665 INFO:teuthology.task.ansible.out:changed: [smithi059.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:16.760157", "end": "2020-03-11 22:27:10.634439", "failed_when_result": false, "rc": 0, "start": "2020-03-11 22:26:53.874282", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2020-03-11T22:27:10.714 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:10 +0000 (0:00:17.036) 0:00:48.796 ******* 2020-03-11T22:27:11.220 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2020-03-11T22:27:11.220 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2020-03-11T22:27:11.258 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:11 +0000 (0:00:00.544) 0:00:49.340 ******* 2020-03-11T22:27:11.480 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2020-03-11T22:27:11.481 INFO:teuthology.task.ansible.out:changed: [smithi059.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.014064", "end": "2020-03-11 22:27:11.452463", "rc": 0, "start": "2020-03-11 22:27:11.438399", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:27:11.519 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:11 +0000 (0:00:00.260) 0:00:49.600 ******* 2020-03-11T22:27:11.743 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2020-03-11T22:27:11.744 INFO:teuthology.task.ansible.out:changed: [smithi059.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.008161", "end": "2020-03-11 22:27:11.723476", "rc": 0, "start": "2020-03-11 22:27:11.715315", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:27:11.782 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:11 +0000 (0:00:00.263) 0:00:49.864 ******* 2020-03-11T22:27:28.680 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2020-03-11T22:27:28.681 INFO:teuthology.task.ansible.out:changed: [smithi059.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:16.684315", "end": "2020-03-11 22:27:28.653228", "rc": 0, "start": "2020-03-11 22:27:11.968913", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:27:28.734 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:28 +0000 (0:00:16.951) 0:01:06.816 ******* 2020-03-11T22:27:28.968 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2020-03-11T22:27:28.970 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2020-03-11T22:27:29.021 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:29 +0000 (0:00:00.287) 0:01:07.103 ******* 2020-03-11T22:27:29.060 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2020-03-11T22:27:29.061 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi", "iozone"], "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/dnf_systems.yml"], "changed": false} 2020-03-11T22:27:29.099 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:29 +0000 (0:00:00.077) 0:01:07.180 ******* 2020-03-11T22:27:29.156 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2020-03-11T22:27:29.157 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-03-11T22:27:29.195 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:29 +0000 (0:00:00.096) 0:01:07.276 ******* 2020-03-11T22:27:29.242 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2020-03-11T22:27:29.244 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml) => {"ansible_facts": {"common_yum_repos": {"lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "copr_repos": ["ktdreyer/ceph-el8"], "epel_packages": [], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["@core", "@base", "dnf-utils", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "podman", "podman-docker", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "dbench", "autoconf", "gdb"], "yum_mirrorlists": ["CentOS-AppStream-mirrorlist", "CentOS-Base-mirrorlist", "CentOS-Extras-mirrorlist"], "yum_repos": {"CentOS-AppStream": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}, "CentOS-Base": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}, "CentOS-Extras": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"} 2020-03-11T22:27:29.305 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:29 +0000 (0:00:00.110) 0:01:07.387 ******* 2020-03-11T22:27:29.355 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2020-03-11T22:27:29.357 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-03-11T22:27:29.403 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:29 +0000 (0:00:00.098) 0:01:07.485 ******* 2020-03-11T22:27:29.668 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2020-03-11T22:27:29.670 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2020-03-11T22:27:29.706 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:29 +0000 (0:00:00.302) 0:01:07.788 ******* 2020-03-11T22:27:29.949 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2020-03-11T22:27:29.951 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => {"changed": false, "gid": 1211, "name": "ubuntu", "state": "present", "system": false} 2020-03-11T22:27:30.000 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:29 +0000 (0:00:00.293) 0:01:08.082 ******* 2020-03-11T22:27:30.026 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2020-03-11T22:27:30.028 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:30.076 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:30 +0000 (0:00:00.075) 0:01:08.157 ******* 2020-03-11T22:27:30.118 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2020-03-11T22:27:30.120 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:30.157 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:30 +0000 (0:00:00.081) 0:01:08.239 ******* 2020-03-11T22:27:30.396 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2020-03-11T22:27:30.398 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2020-03-11T22:27:30.444 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:30 +0000 (0:00:00.286) 0:01:08.526 ******* 2020-03-11T22:27:31.184 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2020-03-11T22:27:31.185 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2020-03-11T22:27:31.223 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:31 +0000 (0:00:00.777) 0:01:09.304 ******* 2020-03-11T22:27:31.259 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2020-03-11T22:27:31.260 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi059"}, "changed": false} 2020-03-11T22:27:31.318 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:31 +0000 (0:00:00.095) 0:01:09.400 ******* 2020-03-11T22:27:32.496 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2020-03-11T22:27:32.497 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "", "ansible_fqdn": "smithi059", "ansible_hostname": "smithi059", "ansible_nodename": "smithi059"}, "changed": false, "name": "smithi059"} 2020-03-11T22:27:32.534 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:32 +0000 (0:00:01.215) 0:01:10.616 ******* 2020-03-11T22:27:33.062 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2020-03-11T22:27:33.062 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2020-03-11T22:27:33.107 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:33 +0000 (0:00:00.572) 0:01:11.188 ******* 2020-03-11T22:27:33.675 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2020-03-11T22:27:33.675 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2020-03-11T22:27:33.717 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:33 +0000 (0:00:00.610) 0:01:11.799 ******* 2020-03-11T22:27:34.313 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2020-03-11T22:27:34.314 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => {"attempts": 1, "changed": false, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2020-03-11T22:27:34.352 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:34 +0000 (0:00:00.635) 0:01:12.434 ******* 2020-03-11T22:27:34.877 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2020-03-11T22:27:34.877 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2020-03-11T22:27:34.914 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:34 +0000 (0:00:00.561) 0:01:12.996 ******* 2020-03-11T22:27:35.131 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** ok: [smithi059.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2020-03-11T22:27:35.165 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:35 +0000 (0:00:00.251) 0:01:13.247 ******* 2020-03-11T22:27:35.388 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-03-11T22:27:35.390 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => {"changed": false, "gid": 36, "name": "kvm", "state": "present", "system": false} 2020-03-11T22:27:35.425 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:35 +0000 (0:00:00.259) 0:01:13.507 ******* 2020-03-11T22:27:35.675 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-03-11T22:27:35.676 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1211, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2020-03-11T22:27:35.714 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:35 +0000 (0:00:00.288) 0:01:13.795 ******* 2020-03-11T22:27:36.240 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-03-11T22:27:36.240 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2020-03-11T22:27:36.280 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:36 +0000 (0:00:00.566) 0:01:14.361 ******* 2020-03-11T22:27:36.824 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-03-11T22:27:36.825 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2020-03-11T22:27:36.864 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:36 +0000 (0:00:00.583) 0:01:14.945 ******* 2020-03-11T22:27:38.861 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-03-11T22:27:38.863 INFO:teuthology.task.ansible.out: [WARNING]: Consider using the yum, dnf or zypper module rather than running 'rpm'. If you need to use command because yum, dnf or zypper is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi059.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:01.781760", "end": "2020-03-11 22:27:38.828200", "rc": 0, "start": "2020-03-11 22:27:37.046440", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:27:38.900 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:38 +0000 (0:00:02.035) 0:01:16.981 ******* 2020-03-11T22:27:39.437 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2020-03-11T22:27:39.439 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => (item=CentOS-AppStream-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "53b24162baf9c2fcf22da2b5c2c258a91997ffad", "dest": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-AppStream-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 425, "state": "file", "uid": 0} 2020-03-11T22:27:39.937 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => (item=CentOS-Base-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "3e4763db3d6c8e2d723452e76e0b9ed9030ed28d", "dest": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Base-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 408, "state": "file", "uid": 0} 2020-03-11T22:27:40.520 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => (item=CentOS-Extras-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "6d4b75787c58f146a27a5ee2b688dd525283abf1", "dest": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Extras-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 410, "state": "file", "uid": 0} 2020-03-11T22:27:40.559 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:40 +0000 (0:00:01.659) 0:01:18.641 ******* 2020-03-11T22:27:41.084 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2020-03-11T22:27:41.085 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => (item={'value': {u'gpgcheck': 0, u'enabled': 1, u'name': u'lab-extras', u'baseurl': u'http://apt-mirror.front.sepia.ceph.com/lab-extras/8/'}, 'key': u'lab-extras'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a0e2ac0628a264e14b96b1c57627a3aa344a6c52", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 203, "state": "file", "uid": 0} 2020-03-11T22:27:41.125 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:41 +0000 (0:00:00.565) 0:01:19.207 ******* 2020-03-11T22:27:41.641 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2020-03-11T22:27:41.643 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => (item={'value': {u'gpgcheck': 1, u'gpgkey': u'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial', u'enabled': 1, u'mirrorlist': u'file:///etc/yum.repos.d/CentOS-Base-mirrorlist', u'name': u'CentOS-$releasever - Base'}, 'key': u'CentOS-Base'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "77e4590ed0ecec056fa44f4e65af60e110822b85", "dest": "/etc/yum.repos.d/CentOS-Base.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Base", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 274, "state": "file", "uid": 0} 2020-03-11T22:27:42.144 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => (item={'value': {u'gpgcheck': 1, u'gpgkey': u'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial', u'enabled': 1, u'mirrorlist': u'file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist', u'name': u'CentOS-$releasever - AppStream'}, 'key': u'CentOS-AppStream'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "9b147c879e95cd23f3bbde027da102febcaad1a8", "dest": "/etc/yum.repos.d/CentOS-AppStream.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-AppStream", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 289, "state": "file", "uid": 0} 2020-03-11T22:27:42.633 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => (item={'value': {u'gpgcheck': 1, u'gpgkey': u'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial', u'enabled': 1, u'mirrorlist': u'file:///etc/yum.repos.d/CentOS-Extras-mirrorlist', u'name': u'CentOS-$releasever - Extras'}, 'key': u'CentOS-Extras'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "2856826614c7f1cb9409228f6c41319a023ca754", "dest": "/etc/yum.repos.d/CentOS-Extras.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Extras", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 280, "state": "file", "uid": 0} 2020-03-11T22:27:42.680 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:42 +0000 (0:00:01.554) 0:01:20.762 ******* 2020-03-11T22:27:43.203 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2020-03-11T22:27:43.205 INFO:teuthology.task.ansible.out:changed: [smithi059.front.sepia.ceph.com] => (item=ktdreyer/ceph-el8) => {"ansible_loop_var": "item", "changed": true, "cmd": ["dnf", "-y", "copr", "enable", "ktdreyer/ceph-el8"], "delta": "0:00:00.295543", "end": "2020-03-11 22:27:43.171108", "item": "ktdreyer/ceph-el8", "rc": 0, "start": "2020-03-11 22:27:42.875565", "stderr": "", "stderr_lines": [], "stdout": "Repository successfully enabled.", "stdout_lines": ["Repository successfully enabled."], "warnings": ["Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message."]} 2020-03-11T22:27:43.206 INFO:teuthology.task.ansible.out: [WARNING]: Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. 2020-03-11T22:27:43.241 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:43 +0000 (0:00:00.561) 0:01:21.323 ******* 2020-03-11T22:27:43.267 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2020-03-11T22:27:43.269 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.304 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:43 +0000 (0:00:00.062) 0:01:21.385 ******* 2020-03-11T22:27:43.342 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2020-03-11T22:27:43.344 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.355 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.391 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:43 +0000 (0:00:00.087) 0:01:21.473 ******* 2020-03-11T22:27:43.428 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2020-03-11T22:27:43.429 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.464 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:43 +0000 (0:00:00.073) 0:01:21.546 ******* 2020-03-11T22:27:43.503 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-03-11T22:27:43.504 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => {"ansible_facts": {"skip_packaging": false}, "changed": false} 2020-03-11T22:27:43.542 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:43 +0000 (0:00:00.078) 0:01:21.624 ******* 2020-03-11T22:27:43.579 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2020-03-11T22:27:43.580 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.616 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:43 +0000 (0:00:00.073) 0:01:21.698 ******* 2020-03-11T22:27:43.653 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2020-03-11T22:27:43.655 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:43.689 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:43 +0000 (0:00:00.072) 0:01:21.771 ******* 2020-03-11T22:27:43.915 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2020-03-11T22:27:43.916 INFO:teuthology.task.ansible.out:fatal: [smithi059.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.006865", "end": "2020-03-11 22:27:43.885462", "msg": "non-zero return code", "rc": 1, "start": "2020-03-11 22:27:43.878597", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2020-03-11T22:27:43.965 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:43 +0000 (0:00:00.276) 0:01:22.047 ******* 2020-03-11T22:27:43.994 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2020-03-11T22:27:43.996 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:27:44.032 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:44 +0000 (0:00:00.066) 0:01:22.113 ******* 2020-03-11T22:27:46.735 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-03-11T22:27:46.736 INFO:teuthology.task.ansible.out:changed: [smithi059.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: qemu-kvm-block-rbd-15:2.12.0-88.module_el8.1.0+266+ba744077.2.x86_64", "Removed: librados2-1:12.2.7-9.el8.x86_64", "Removed: librbd1-1:12.2.7-9.el8.x86_64"]} 2020-03-11T22:27:46.798 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:46 +0000 (0:00:02.766) 0:01:24.880 ******* 2020-03-11T22:27:49.986 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-03-11T22:27:49.987 INFO:teuthology.task.ansible.out:changed: [smithi059.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: boost-program-options-1.66.0-6.el8.x86_64", "Removed: boost-random-1.66.0-6.el8.x86_64", "Removed: hdparm-9.54-2.el8.x86_64", "Removed: tuned-2.12.0-3.el8_1.1.noarch"]} 2020-03-11T22:27:50.024 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:50 +0000 (0:00:03.225) 0:01:28.105 ******* 2020-03-11T22:27:55.208 INFO:teuthology.task.ansible.out: TASK [testnode : Base group workaround for BZ1782899] ************************** 2020-03-11T22:27:55.209 INFO:teuthology.task.ansible.out:changed: [smithi059.front.sepia.ceph.com] => {"changed": true, "cmd": "dnf -y groupinstall base --nobest && dnf -y group upgrade base", "delta": "0:00:04.901761", "end": "2020-03-11 22:27:55.183121", "rc": 0, "start": "2020-03-11 22:27:50.281360", "stderr": "No match for group package \"insights-client\"\nNo match for group package \"centos-release-eula\"\nNo match for group package \"insights-client\"\nNo match for group package \"centos-release-eula\"", "stderr_lines": ["No match for group package \"insights-client\"", "No match for group package \"centos-release-eula\"", "No match for group package \"insights-client\"", "No match for group package \"centos-release-eula\""], "stdout": "CentOS-8 - AppStream 141 kB/s | 4.3 kB 00:00 \nCentOS-8 - Base 129 kB/s | 3.8 kB 00:00 \nCentOS-8 - Extras 51 kB/s | 1.5 kB 00:00 \nCopr repo for ceph-el8 owned by ktdreyer 69 kB/s | 3.3 kB 00:00 \nExtra Packages for Enterprise Linux 138 kB/s | 5.3 kB 00:00 \nlab-extras 1.2 MB/s | 2.9 kB 00:00 \nDependencies resolved.\n================================================================================\n Package Architecture Version Repository Size\n================================================================================\nInstalling Groups:\n Base \n\nTransaction Summary\n================================================================================\n\nComplete!\nLast metadata expiration check: 0:00:03 ago on Wed 11 Mar 2020 10:27:50 PM UTC.\nDependencies resolved.\n================================================================================\n Package Architecture Version Repository Size\n============================================================= 2020-03-11T22:27:55.210 INFO:teuthology.task.ansible.out:===================\nUpgrading Groups:\n Base \n\nTransaction Summary\n================================================================================\n\nComplete!", "stdout_lines": ["CentOS-8 - AppStream 141 kB/s | 4.3 kB 00:00 ", "CentOS-8 - Base 129 kB/s | 3.8 kB 00:00 ", "CentOS-8 - Extras 51 kB/s | 1.5 kB 00:00 ", "Copr repo for ceph-el8 owned by ktdreyer 69 kB/s | 3.3 kB 00:00 ", "Extra Packages for Enterprise Linux 138 kB/s | 5.3 kB 00:00 ", "lab-extras 1.2 MB/s | 2.9 kB 00:00 ", "Dependencies resolved.", "================================================================================", " Package Architecture Version Repository Size", "================================================================================", "Installing Groups:", " Base ", "", "Transaction Summary", "================================================================================", "", "Complete!", "Last metadata expiration check: 0:00:03 ago on Wed 11 Mar 2020 10:27:50 PM UTC.", "Dependencies resolved.", "================================================================================", " Package Architecture Version Repository Size", "================================================================================", "Upgrading Groups:", " Base ", "", "Transaction Summary", "================================================================================", "", "Complete!"]} 2020-03-11T22:27:55.249 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:27:55 +0000 (0:00:05.225) 0:01:33.331 ******* 2020-03-11T22:28:03.038 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-03-11T22:28:03.038 INFO:teuthology.task.ansible.out:changed: [smithi059.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Group core installed.", "Group base installed.", "Installed: dnf-utils", "Installed: sysstat", "Installed: libedit", "Installed: boost-thread", "Installed: xfsprogs", "Installed: gdisk", "Installed: parted", "Installed: libgcrypt", "Installed: fuse-libs", "Installed: openssl", "Installed: libuuid", "Installed: podman", "Installed: podman-docker", "Installed: attr", "Installed: ant", "Installed: lsof", "Installed: gettext", "Installed: bc", "Installed: xfsdump", "Installed: blktrace", "Installed: usbredir", "Installed: libev-devel", "Installed: valgrind", "Installed: nfs-utils", "Installed: ncurses-devel", "Installed: gcc", "Installed: git", "Installed: python3-nose", "Installed: python3-virtualenv", "Installed: genisoimage", "Installed: qemu-img", "Installed: qemu-kvm-core", "Installed: qemu-kvm-block-rbd", "Installed: libacl-devel", "Installed: dbench", "Installed: autoconf", "Installed: gdb", "Installed: tuned-2. 2020-03-11T22:28:03.039 INFO:teuthology.task.ansible.out:12.0-3.el8_1.1.noarch", "Installed: librados2-1:12.2.7-9.el8.x86_64", "Installed: librbd1-1:12.2.7-9.el8.x86_64", "Installed: qemu-kvm-block-rbd-15:2.12.0-88.module_el8.1.0+266+ba744077.2.x86_64", "Installed: boost-program-options-1.66.0-6.el8.x86_64", "Installed: boost-random-1.66.0-6.el8.x86_64", "Installed: hdparm-9.54-2.el8.x86_64"]} 2020-03-11T22:28:03.076 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:03 +0000 (0:00:07.826) 0:01:41.157 ******* 2020-03-11T22:28:03.112 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2020-03-11T22:28:03.113 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:03.152 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:03 +0000 (0:00:00.076) 0:01:41.234 ******* 2020-03-11T22:28:04.145 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-03-11T22:28:04.147 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2020-03-11T22:28:04.201 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:04 +0000 (0:00:01.048) 0:01:42.282 ******* 2020-03-11T22:28:04.238 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-03-11T22:28:04.240 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:04.279 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:04 +0000 (0:00:00.077) 0:01:42.360 ******* 2020-03-11T22:28:04.634 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2020-03-11T22:28:04.637 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "sysinit.target dbus.socket system.slice dbus.service basic.target polkit.service", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target network-pre.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "ip6tables.service iptables.service ebtables.service shutdown.target ipset.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "firewalld - dynamic firewall daemon", "DevicePolicy": "auto", "Documentation": "man:firewalld(1)", "DynamicUser": "no", "EnvironmentFiles": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTim 2020-03-11T22:28:04.638 INFO:teuthology.task.ansible.out:estampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "firewalld.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "127467", "LimitNPROCSoft": "127467", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127467", 2020-03-11T22:28:04.640 INFO:teuthology.task.ansible.out:"LimitSIGPENDINGSoft": "127467", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "yes", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "dbus.socket sysinit.target system.slice", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "null", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestampMonotonic": "0", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "SyslogFacility": "3 2020-03-11T22:28:04.641 INFO:teuthology.task.ansible.out:", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "26213", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "network-pre.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2020-03-11T22:28:04.682 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:04 +0000 (0:00:00.403) 0:01:42.764 ******* 2020-03-11T22:28:05.000 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ fatal: [smithi059.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2020-03-11T22:28:05.045 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:05 +0000 (0:00:00.362) 0:01:43.126 ******* 2020-03-11T22:28:05.992 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2020-03-11T22:28:05.993 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2020-03-11T22:28:06.042 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:06 +0000 (0:00:00.997) 0:01:44.124 ******* 2020-03-11T22:28:06.068 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2020-03-11T22:28:06.070 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:06.106 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:06 +0000 (0:00:00.063) 0:01:44.188 ******* 2020-03-11T22:28:06.133 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2020-03-11T22:28:06.182 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:06 +0000 (0:00:00.075) 0:01:44.264 ******* 2020-03-11T22:28:06.210 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2020-03-11T22:28:06.211 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:06.258 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:06 +0000 (0:00:00.076) 0:01:44.340 ******* 2020-03-11T22:28:06.286 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2020-03-11T22:28:06.288 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:06.330 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:06 +0000 (0:00:00.071) 0:01:44.411 ******* 2020-03-11T22:28:06.369 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2020-03-11T22:28:06.371 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2020-03-11T22:28:06.380 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2020-03-11T22:28:06.418 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:06 +0000 (0:00:00.087) 0:01:44.499 ******* 2020-03-11T22:28:06.444 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:06.481 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:06 +0000 (0:00:00.063) 0:01:44.563 ******* 2020-03-11T22:28:06.508 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2020-03-11T22:28:06.546 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:06 +0000 (0:00:00.065) 0:01:44.628 ******* 2020-03-11T22:28:06.583 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2020-03-11T22:28:06.584 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:06.620 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:06 +0000 (0:00:00.073) 0:01:44.702 ******* 2020-03-11T22:28:06.646 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2020-03-11T22:28:06.650 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:06.686 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:06 +0000 (0:00:00.065) 0:01:44.768 ******* 2020-03-11T22:28:06.712 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2020-03-11T22:28:06.714 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:06.750 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:06 +0000 (0:00:00.063) 0:01:44.831 ******* 2020-03-11T22:28:06.787 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-03-11T22:28:06.788 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:06.833 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:06 +0000 (0:00:00.083) 0:01:44.915 ******* 2020-03-11T22:28:06.862 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2020-03-11T22:28:06.864 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:06.907 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:06 +0000 (0:00:00.073) 0:01:44.988 ******* 2020-03-11T22:28:06.944 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:06.993 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:06 +0000 (0:00:00.086) 0:01:45.075 ******* 2020-03-11T22:28:07.022 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2020-03-11T22:28:07.024 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:07.060 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:07 +0000 (0:00:00.066) 0:01:45.141 ******* 2020-03-11T22:28:07.098 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2020-03-11T22:28:07.100 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:07.150 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:07 +0000 (0:00:00.090) 0:01:45.232 ******* 2020-03-11T22:28:07.187 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2020-03-11T22:28:07.189 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:07.229 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:07 +0000 (0:00:00.079) 0:01:45.311 ******* 2020-03-11T22:28:07.255 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2020-03-11T22:28:07.257 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:07.299 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:07 +0000 (0:00:00.069) 0:01:45.381 ******* 2020-03-11T22:28:07.326 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2020-03-11T22:28:07.327 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:07.368 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:07 +0000 (0:00:00.068) 0:01:45.449 ******* 2020-03-11T22:28:07.404 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:07.445 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:07 +0000 (0:00:00.077) 0:01:45.527 ******* 2020-03-11T22:28:07.474 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2020-03-11T22:28:07.475 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:07.518 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:07 +0000 (0:00:00.073) 0:01:45.600 ******* 2020-03-11T22:28:07.555 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2020-03-11T22:28:07.556 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:07.599 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:07 +0000 (0:00:00.080) 0:01:45.680 ******* 2020-03-11T22:28:07.625 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-03-11T22:28:07.627 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:07.664 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:07 +0000 (0:00:00.064) 0:01:45.745 ******* 2020-03-11T22:28:07.700 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-03-11T22:28:07.702 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:07.741 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:07 +0000 (0:00:00.077) 0:01:45.823 ******* 2020-03-11T22:28:07.768 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-03-11T22:28:07.770 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:07.807 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:07 +0000 (0:00:00.065) 0:01:45.888 ******* 2020-03-11T22:28:07.835 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-03-11T22:28:07.837 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:07.892 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:07 +0000 (0:00:00.084) 0:01:45.973 ******* 2020-03-11T22:28:07.918 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-03-11T22:28:07.920 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:07.957 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:07 +0000 (0:00:00.065) 0:01:46.039 ******* 2020-03-11T22:28:07.983 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-03-11T22:28:07.984 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:08.023 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:08 +0000 (0:00:00.065) 0:01:46.104 ******* 2020-03-11T22:28:08.048 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-03-11T22:28:08.049 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:08.088 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:08 +0000 (0:00:00.065) 0:01:46.170 ******* 2020-03-11T22:28:08.116 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-03-11T22:28:08.118 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:08.157 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:08 +0000 (0:00:00.068) 0:01:46.238 ******* 2020-03-11T22:28:08.182 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-03-11T22:28:08.184 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:08.234 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:08 +0000 (0:00:00.077) 0:01:46.316 ******* 2020-03-11T22:28:08.261 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-03-11T22:28:08.262 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:08.305 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:08 +0000 (0:00:00.070) 0:01:46.386 ******* 2020-03-11T22:28:08.332 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-03-11T22:28:08.333 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:08.375 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:08 +0000 (0:00:00.070) 0:01:46.457 ******* 2020-03-11T22:28:08.402 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-03-11T22:28:08.403 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:08.443 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:08 +0000 (0:00:00.068) 0:01:46.525 ******* 2020-03-11T22:28:08.679 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** ok: [smithi059.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/bin/lspci", "gid": 0, "group": "root", "mode": "0777", "owner": "root", "secontext": "unconfined_u:object_r:bin_t:s0", "size": 11, "src": "/sbin/lspci", "state": "link", "uid": 0} 2020-03-11T22:28:08.717 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:08 +0000 (0:00:00.273) 0:01:46.799 ******* 2020-03-11T22:28:08.939 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-03-11T22:28:08.941 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/ltp/testcases/bin", "secontext": "unconfined_u:object_r:lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2020-03-11T22:28:08.978 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:08 +0000 (0:00:00.260) 0:01:47.059 ******* 2020-03-11T22:28:09.193 INFO:teuthology.task.ansible.out: 2020-03-11T22:28:09.195 INFO:teuthology.task.ansible.out:TASK [testnode : Make fsstress same path as ubuntu.] *************************** [WARNING]: Cannot set fs attributes on a non-existent symlink target. follow should be set to False to avoid this. ok: [smithi059.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/lib/ltp/testcases/bin/fsstress", "src": "/usr/bin/fsstress"} 2020-03-11T22:28:09.231 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:09 +0000 (0:00:00.253) 0:01:47.313 ******* 2020-03-11T22:28:09.259 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-03-11T22:28:09.260 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:09.297 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:09 +0000 (0:00:00.065) 0:01:47.379 ******* 2020-03-11T22:28:09.325 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-03-11T22:28:09.326 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:09.365 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:09 +0000 (0:00:00.067) 0:01:47.447 ******* 2020-03-11T22:28:09.391 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:09.429 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:09 +0000 (0:00:00.063) 0:01:47.510 ******* 2020-03-11T22:28:09.470 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-03-11T22:28:09.470 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:09.510 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:09 +0000 (0:00:00.081) 0:01:47.592 ******* 2020-03-11T22:28:09.544 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-03-11T22:28:09.544 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:09.584 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:09 +0000 (0:00:00.074) 0:01:47.666 ******* 2020-03-11T22:28:09.611 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-03-11T22:28:09.612 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:09.677 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:09 +0000 (0:00:00.092) 0:01:47.758 ******* 2020-03-11T22:28:09.706 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-03-11T22:28:09.707 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:09.751 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:09 +0000 (0:00:00.074) 0:01:47.833 ******* 2020-03-11T22:28:09.778 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-03-11T22:28:09.778 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:09.780 INFO:teuthology.task.ansible.out: 2020-03-11T22:28:09.817 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:09 +0000 (0:00:00.065) 0:01:47.898 ******* 2020-03-11T22:28:09.818 INFO:teuthology.task.ansible.out: 2020-03-11T22:28:09.854 INFO:teuthology.task.ansible.out: 2020-03-11T22:28:09.856 INFO:teuthology.task.ansible.out:TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:09.898 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:09 +0000 (0:00:00.081) 0:01:47.980 ******* 2020-03-11T22:28:09.898 INFO:teuthology.task.ansible.out: 2020-03-11T22:28:09.925 INFO:teuthology.task.ansible.out: 2020-03-11T22:28:09.927 INFO:teuthology.task.ansible.out:TASK [testnode : Set grub config.] ********************************************* skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:09.965 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:09 +0000 (0:00:00.067) 0:01:48.047 ******* 2020-03-11T22:28:10.138 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2020-03-11T22:28:10.139 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:10.285 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:10 +0000 (0:00:00.319) 0:01:48.367 ******* 2020-03-11T22:28:10.312 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-03-11T22:28:10.313 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:10.355 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:10 +0000 (0:00:00.069) 0:01:48.437 ******* 2020-03-11T22:28:10.383 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-03-11T22:28:10.383 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:10.422 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:10 +0000 (0:00:00.067) 0:01:48.504 ******* 2020-03-11T22:28:10.459 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2020-03-11T22:28:10.460 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:10.498 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:10 +0000 (0:00:00.076) 0:01:48.580 ******* 2020-03-11T22:28:10.527 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2020-03-11T22:28:10.528 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:10.585 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:10 +0000 (0:00:00.086) 0:01:48.667 ******* 2020-03-11T22:28:10.612 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2020-03-11T22:28:10.612 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:10.655 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:10 +0000 (0:00:00.069) 0:01:48.736 ******* 2020-03-11T22:28:10.681 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2020-03-11T22:28:10.683 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:10.722 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:10 +0000 (0:00:00.067) 0:01:48.804 ******* 2020-03-11T22:28:10.748 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2020-03-11T22:28:10.750 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:10.788 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:10 +0000 (0:00:00.065) 0:01:48.870 ******* 2020-03-11T22:28:10.815 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2020-03-11T22:28:10.817 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:10.863 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:10 +0000 (0:00:00.074) 0:01:48.945 ******* 2020-03-11T22:28:10.889 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2020-03-11T22:28:10.890 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:10.925 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:10 +0000 (0:00:00.062) 0:01:49.007 ******* 2020-03-11T22:28:10.951 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2020-03-11T22:28:10.953 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:10.988 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:10 +0000 (0:00:00.063) 0:01:49.070 ******* 2020-03-11T22:28:11.014 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2020-03-11T22:28:11.016 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:11.065 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:11 +0000 (0:00:00.076) 0:01:49.147 ******* 2020-03-11T22:28:11.092 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-03-11T22:28:11.093 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2020-03-11T22:28:11.133 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:11 +0000 (0:00:00.067) 0:01:49.215 ******* 2020-03-11T22:28:11.182 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2020-03-11T22:28:11.184 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2020-03-11T22:28:11.204 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2020-03-11T22:28:11.205 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2020-03-11T22:28:11.216 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2020-03-11T22:28:11.227 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2020-03-11T22:28:11.238 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2020-03-11T22:28:11.249 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2020-03-11T22:28:11.307 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:11 +0000 (0:00:00.173) 0:01:49.388 ******* 2020-03-11T22:28:11.481 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2020-03-11T22:28:11.482 INFO:teuthology.task.ansible.out:changed: [smithi059.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.009743", "end": "2020-03-11 22:28:11.463102", "rc": 0, "start": "2020-03-11 22:28:11.453359", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2020-03-11T22:28:11.541 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:11 +0000 (0:00:00.234) 0:01:49.622 ******* 2020-03-11T22:28:11.567 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2020-03-11T22:28:11.568 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:11.607 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:11 +0000 (0:00:00.065) 0:01:49.688 ******* 2020-03-11T22:28:11.632 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2020-03-11T22:28:11.634 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:11.674 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:11 +0000 (0:00:00.067) 0:01:49.756 ******* 2020-03-11T22:28:11.711 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2020-03-11T22:28:11.712 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:11.752 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:11 +0000 (0:00:00.078) 0:01:49.834 ******* 2020-03-11T22:28:11.780 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:11.815 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:11 +0000 (0:00:00.063) 0:01:49.897 ******* 2020-03-11T22:28:12.828 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2020-03-11T22:28:12.830 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: gdisk", "Installed: device-mapper"]} 2020-03-11T22:28:12.874 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:12 +0000 (0:00:01.058) 0:01:50.955 ******* 2020-03-11T22:28:12.910 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-03-11T22:28:12.912 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => (item={u'block_used': 12960489, u'uuid': u'ebbdc914-4a8c-428b-9507-ff6e0d466a46', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 227134619, u'size_available': 930343399424, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60989856, u'device': u'/dev/sda1', u'inode_used': 65120, u'block_size': 4096, u'options': u'rw,seclabel,relatime'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 227134619, "block_size": 4096, "block_total": 240095108, "block_used": 12960489, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60989856, "inode_total": 61054976, "inode_used": 65120, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930343399424, "size_total": 983429562368, "uuid": "ebbdc914-4a8c-428b-9507-ff6e0d466a46"}} 2020-03-11T22:28:12.956 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:12 +0000 (0:00:00.082) 0:01:51.038 ******* 2020-03-11T22:28:13.171 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2020-03-11T22:28:13.173 INFO:teuthology.task.ansible.out:changed: [smithi059.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.005461", "end": "2020-03-11 22:28:13.143656", "rc": 0, "start": "2020-03-11 22:28:13.138195", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:28:13.213 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:13 +0000 (0:00:00.254) 0:01:51.294 ******* 2020-03-11T22:28:13.263 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2020-03-11T22:28:13.264 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'block_used': 12960489, u'uuid': u'ebbdc914-4a8c-428b-9507-ff6e0d466a46', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 227134619, u'size_available': 930343399424, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60989856, u'device': u'/dev/sda1', u'inode_used': 65120, u'block_size': 4096, u'options': u'rw,seclabel,relatime'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227134619, "block_size": 4096, "block_total": 240095108, "block_used": 12960489, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60989856, "inode_total": 61054976, "inode_used": 65120, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930343399424, "size_total": 983429562368, "uuid": "ebbdc914-4a8c-428b-9507-ff6e0d466a46"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:13.306 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:13 +0000 (0:00:00.092) 0:01:51.387 ******* 2020-03-11T22:28:13.332 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2020-03-11T22:28:13.370 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:13 +0000 (0:00:00.064) 0:01:51.452 ******* 2020-03-11T22:28:13.418 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-03-11T22:28:13.419 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'ATA', u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W4J2P7', u'wwn-0x5000c500871b7b3d'], u'uuids': []}, u'sas_device_handle': None, u'host': u'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', u'support_discard': u'0', u'serial': u'Z1W4J2P7', u'wwn': u'0x5000c500871b7b3d', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'mq-deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST1000NM0033-9ZM', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'ebbdc914-4a8c-428b-9507-ff6e0d466a46', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W4J2P7-part1', u'wwn-0x5000c500871b7b3d-part1'], u'uuids': [u'ebbdc914-4a8c-428b-9507-ff6e0d466a46']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4J2P7", "wwn-0x5000c500871b7b3d"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4J2P7-part1", "wwn-0x5000c500871b7b3d-part1"], "labels": [], "masters": [], "uuids": ["ebbdc914-4a8c-428b-9507-ff6e0d466a46"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "ebbdc914-4a8c-428b-9507-ff6e0d466a46"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4J2P7", "size": "931.51 GB", "support_discard": "0", "vend 2020-03-11T22:28:13.421 INFO:teuthology.task.ansible.out:or": "ATA", "virtual": 1, "wwn": "0x5000c500871b7b3d"}}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:14.973 INFO:teuthology.task.ansible.out:changed: [smithi059.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [u'dm-0', u'dm-1', u'dm-2', u'dm-3', u'dm-4'], u'labels': [], u'ids': [u'lvm-pv-uuid-mv6lAs-JCnc-HIuG-toO7-jC9s-wXr2-afYfhf', u'nvme-INTEL_SSDPEDMD400G4_CVFT5330009L400BGN', u'nvme-nvme.8086-43564654353333303030394c34303042474e-494e54454c205353445045444d443430304734-00000001'], u'uuids': []}, u'sas_device_handle': None, u'host': u'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', u'support_discard': u'512', u'model': u'INTEL SSDPEDMD400G4', u'size': u'372.61 GB', u'scheduler_mode': u'none', u'rotational': u'0', u'sectors': u'781422768', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [u'vg_nvme-lv_4', u'vg_nvme-lv_1', u'vg_nvme-lv_3', u'vg_nvme-lv_5', u'vg_nvme-lv_2'], u'partitions': {}}, 'key': u'nvme0n1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.335285", "end": "2020-03-11 22:28:14.936796", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_4", "vg_nvme-lv_1", "vg_nvme-lv_3", "vg_nvme-lv_5", "vg_nvme-lv_2"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-mv6lAs-JCnc-HIuG-toO7-jC9s-wXr2-afYfhf", "nvme-INTEL_SSDPEDMD400G4_CVFT5330009L400BGN", "nvme-nvme.8086-43564654353333303030394c34303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:28:13.601511", "stderr": "", "stderr_lines": [], "stdout": "Creating new GP 2020-03-11T22:28:14.974 INFO:teuthology.task.ansible.out:T entries.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-11T22:28:16.232 INFO:teuthology.task.ansible.out:changed: [smithi059.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_1', u'dm-uuid-LVM-gyFGOmnxN83Gg444j7oPCyxE8fLdW2vekcr29hd2d6R70FIMceaLzi9o6khLaHef'], u'uuids': [u'b3916b01-4f33-4472-8c90-431854dfa99a']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-4'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.052142", "end": "2020-03-11 22:28:16.199375", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-gyFGOmnxN83Gg444j7oPCyxE8fLdW2vekcr29hd2d6R70FIMceaLzi9o6khLaHef"], "labels": [], "masters": [], "uuids": ["b3916b01-4f33-4472-8c90-431854dfa99a"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:28:15.147233", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-11T22:28:17.516 INFO:teuthology.task.ansible.out:changed: [smithi059.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_3', u'dm-uuid-LVM-gyFGOmnxN83Gg444j7oPCyxE8fLdW2vehGegzxH3RapXG87yzFc3VoD1Znn6vT8e'], u'uuids': [u'b729efdd-3241-4b55-a2aa-4d15d1b0f764']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-2'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.040115", "end": "2020-03-11 22:28:17.465133", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-gyFGOmnxN83Gg444j7oPCyxE8fLdW2vehGegzxH3RapXG87yzFc3VoD1Znn6vT8e"], "labels": [], "masters": [], "uuids": ["b729efdd-3241-4b55-a2aa-4d15d1b0f764"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:28:16.425018", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-11T22:28:18.757 INFO:teuthology.task.ansible.out:changed: [smithi059.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_2', u'dm-uuid-LVM-gyFGOmnxN83Gg444j7oPCyxE8fLdW2veGFdDuuksM50bItRXW0Exzxz1BFW8t3TA'], u'uuids': [u'6fab08c6-7d63-40a4-a48b-75833858c249']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-3'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.044174", "end": "2020-03-11 22:28:18.724154", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-gyFGOmnxN83Gg444j7oPCyxE8fLdW2veGFdDuuksM50bItRXW0Exzxz1BFW8t3TA"], "labels": [], "masters": [], "uuids": ["6fab08c6-7d63-40a 2020-03-11T22:28:18.759 INFO:teuthology.task.ansible.out:4-a48b-75833858c249"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:28:17.679980", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-11T22:28:20.030 INFO:teuthology.task.ansible.out:changed: [smithi059.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_5', u'dm-uuid-LVM-gyFGOmnxN83Gg444j7oPCyxE8fLdW2vemDlX937t8kGJ14cOqCQiF3zkdgm2OMd6'], u'uuids': [u'b69b1704-cb63-4902-b12b-3cf1eda8cf34']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'14.90 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'31252480', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-0'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.057828", "end": "2020-03-11 22:28:19.999228", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-gyFGOmnxN83Gg444j7oPCyxE8fLdW2vemDlX937t8kGJ14cOqCQiF3zkdgm2OMd6"], "labels": [], "masters": [], "uuids": ["b69b1704-cb63-4902-b12b-3cf1eda8cf34"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:28:18.941400", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-11T22:28:21.296 INFO:teuthology.task.ansible.out:changed: [smithi059.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_4', u'dm-uuid-LVM-gyFGOmnxN83Gg444j7oPCyxE8fLdW2vebR0PaApff8fkNTry9qDJOYmpZcf07paZ'], u'uuids': [u'a65947b3-b0aa-47fc-9a65-e0d33aab8bd9']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.051566", "end": "2020-03-11 22:28:21.258108", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-gyFGOmnxN83Gg444j7oPCyxE8fLdW2vebR0PaApff8fkNTry9qDJOYmpZcf07paZ"], "labels": [], "masters": [], "uuids": ["a65947b3-b0aa-47fc-9a65-e0d33aab8bd9"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:28:20.206542", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-11T22:28:21.342 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:21 +0000 (0:00:07.971) 0:01:59.423 ******* 2020-03-11T22:28:21.393 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-03-11T22:28:21.394 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'ATA', u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W4J2P7', u'wwn-0x5000c500871b7b3d'], u'uuids': []}, u'sas_device_handle': None, u'host': u'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', u'support_discard': u'0', u'serial': u'Z1W4J2P7', u'wwn': u'0x5000c500871b7b3d', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'mq-deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST1000NM0033-9ZM', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'ebbdc914-4a8c-428b-9507-ff6e0d466a46', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W4J2P7-part1', u'wwn-0x5000c500871b7b3d-part1'], u'uuids': [u'ebbdc914-4a8c-428b-9507-ff6e0d466a46']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4J2P7", "wwn-0x5000c500871b7b3d"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4J2P7-part1", "wwn-0x5000c500871b7b3d-part1"], "labels": [], "masters": [], "uuids": ["ebbdc914-4a8c-428b-9507-ff6e0d466a46"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "ebbdc914-4a8c-428b-9507-ff6e0d466a46"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4J2P7", "size": "931.51 GB", "support_discard": "0", "vend 2020-03-11T22:28:21.395 INFO:teuthology.task.ansible.out:or": "ATA", "virtual": 1, "wwn": "0x5000c500871b7b3d"}}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:21.647 INFO:teuthology.task.ansible.out:changed: [smithi059.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [u'dm-0', u'dm-1', u'dm-2', u'dm-3', u'dm-4'], u'labels': [], u'ids': [u'lvm-pv-uuid-mv6lAs-JCnc-HIuG-toO7-jC9s-wXr2-afYfhf', u'nvme-INTEL_SSDPEDMD400G4_CVFT5330009L400BGN', u'nvme-nvme.8086-43564654353333303030394c34303042474e-494e54454c205353445045444d443430304734-00000001'], u'uuids': []}, u'sas_device_handle': None, u'host': u'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', u'support_discard': u'512', u'model': u'INTEL SSDPEDMD400G4', u'size': u'372.61 GB', u'scheduler_mode': u'none', u'rotational': u'0', u'sectors': u'781422768', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [u'vg_nvme-lv_4', u'vg_nvme-lv_1', u'vg_nvme-lv_3', u'vg_nvme-lv_5', u'vg_nvme-lv_2'], u'partitions': {}}, 'key': u'nvme0n1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.036 2020-03-11T22:28:21.649 INFO:teuthology.task.ansible.out:508", "end": "2020-03-11 22:28:21.616710", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_4", "vg_nvme-lv_1", "vg_nvme-lv_3", "vg_nvme-lv_5", "vg_nvme-lv_2"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-mv6lAs-JCnc-HIuG-toO7-jC9s-wXr2-afYfhf", "nvme-INTEL_SSDPEDMD400G4_CVFT5330009L400BGN", "nvme-nvme.8086-43564654353333303030394c34303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:28:21.580202", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0338209 s, 3.4 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0338209 s, 3.4 GB/s"], "stdout": "", "stdout_lines": []} 2020-03-11T22:28:22.001 INFO:teuthology.task.ansible.out:changed: [smithi059.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_1', u'dm-uuid-LVM-gyFGOmnxN83Gg444j7oPCyxE8fLdW2vekcr29hd2d6R70FIMceaLzi9o6khLaHef'], u'uuids': [u'b3916b01-4f33-4472-8c90-431854dfa99a']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-4'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.139375", "end": "2020-03-11 22:28:21.964025", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-gyFGOmnxN83Gg444j7oPCyxE8fLdW2vekcr29hd2d6R70FIMceaLzi9o6khLaHef"], "labels": [], "masters": [], "uuids": ["b3916b01-4f33-4472-8c90-431854dfa99a"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:28:21.824650", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.136602 s, 844 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.136602 s, 844 MB/s"], "stdout": "", "stdout_lines": []} 2020-03-11T22:28:22.334 INFO:teuthology.task.ansible.out:changed: [smithi059.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_3', u'dm-uuid-LVM-gyFGOmnxN83Gg444j7oPCyxE8fLdW2vehGegzxH3RapXG87yzFc3VoD1Znn6vT8e'], u'uuids': [u'b729efdd-3241-4b55-a2aa-4d15d1b0f764']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-2'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.137999", "end": "2020-03-11 22:28:22.300679", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-gyFGOmnxN83Gg444j7oPCyxE8fLdW2vehGegzxH3RapXG87yzFc3VoD1Znn6vT8e"], "labels": [], "masters": [], "uuids": ["b729efdd-3241-4b55-a2aa-4d15d1b0f764"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:28:22.162680", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.13535 s, 852 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.13535 s, 852 MB/s"], "stdout": "", "stdout_lines": []} 2020-03-11T22:28:22.668 INFO:teuthology.task.ansible.out:changed: [smithi059.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_2', u'dm-uuid-LVM-gyFGOmnxN83Gg444j7oPCyxE8fLdW2veGFdDuuksM50bItRXW0Exzxz1BFW8t3TA'], u'uuids': [u'6fab08c6-7d63-40a4-a48b-75833858c249']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-3'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.137024", "end": "2020-03-11 22:28:22.635541", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-gyFGOmnxN83Gg444j7oPCyxE8fLdW2veGFdDuuksM50bItRXW0Exzxz1BFW8t3TA"], "labels": [], "masters": [], "uuids": ["6fab08c6-7d63-40a4-a48b-75833858c249"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:28:22.498517", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.134368 s, 858 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.134368 s, 858 MB/s"], "stdout": "", "stdout_lines": []} 2020-03-11T22:28:23.012 INFO:teuthology.task.ansible.out:changed: [smithi059.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_5', u'dm-uuid-LVM-gyFGOmnxN83Gg444j7oPCyxE8fLdW2vemDlX937t8kGJ14cOqCQiF3zkdgm2OMd6'], u'uuids': [u'b69b1704-cb63-4902-b12b-3cf1eda8cf34']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'14.90 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'31252480', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-0'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.151173", "end": "2020-03-11 22:28:22.983944", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-gyFGOmnxN83Gg444j7oPCyxE8fLdW2vemDlX937t8kGJ14cOqCQiF3zkdgm2OMd6"], "labels": [], "masters": [], "uuids": ["b69b1704-cb63-4902-b12b-3cf1eda8cf34"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:28:22.832771", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.148474 s, 777 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.148474 s, 777 MB/s"], "stdout": "", "stdout_lines": []} 2020-03-11T22:28:23.364 INFO:teuthology.task.ansible.out:changed: [smithi059.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_4', u'dm-uuid-LVM-gyFGOmnxN83Gg444j7oPCyxE8fLdW2vebR0PaApff8fkNTry9qDJOYmpZcf07paZ'], u'uuids': [u'a65947b3-b0aa-47fc-9a65-e0d33aab8bd9']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.137893", "end": "2020-03-11 22:28:23.331940", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-gyFGOmnxN83Gg444j7oPCyxE8fLdW2vebR0PaApff8fkNTry9qDJOYmpZcf07paZ"], "labels": [], "masters": [], "uuids": ["a65947b3-b0aa-47fc-9a65-e0d33aab8bd9"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:28:23.194047", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.135221 s, 853 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.135221 s, 853 MB/s"], "stdout": "", "stdout_lines": []} 2020-03-11T22:28:23.408 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:23 +0000 (0:00:02.065) 0:02:01.489 ******* 2020-03-11T22:28:23.682 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2020-03-11T22:28:23.683 INFO:teuthology.task.ansible.out:changed: [smithi059.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.065527", "end": "2020-03-11 22:28:23.653383", "rc": 0, "start": "2020-03-11 22:28:23.587856", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:28:23.720 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:23 +0000 (0:00:00.312) 0:02:01.802 ******* 2020-03-11T22:28:23.973 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2020-03-11T22:28:23.975 INFO:teuthology.task.ansible.out:changed: [smithi059.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.041728", "end": "2020-03-11 22:28:23.948101", "rc": 0, "start": "2020-03-11 22:28:23.906373", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:28:24.012 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:24 +0000 (0:00:00.291) 0:02:02.094 ******* 2020-03-11T22:28:24.038 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2020-03-11T22:28:24.092 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:24 +0000 (0:00:00.080) 0:02:02.174 ******* 2020-03-11T22:28:24.118 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2020-03-11T22:28:24.120 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:24.159 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:24 +0000 (0:00:00.066) 0:02:02.240 ******* 2020-03-11T22:28:24.185 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2020-03-11T22:28:24.186 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:24.223 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:24 +0000 (0:00:00.063) 0:02:02.304 ******* 2020-03-11T22:28:24.249 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-03-11T22:28:24.250 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:24.289 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:24 +0000 (0:00:00.066) 0:02:02.371 ******* 2020-03-11T22:28:24.317 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2020-03-11T22:28:24.319 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:24.355 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:24 +0000 (0:00:00.065) 0:02:02.436 ******* 2020-03-11T22:28:24.568 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2020-03-11T22:28:24.569 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-03-11T22:28:24.606 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:24 +0000 (0:00:00.251) 0:02:02.688 ******* 2020-03-11T22:28:24.643 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-03-11T22:28:24.644 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={u'block_used': 12960489, u'uuid': u'ebbdc914-4a8c-428b-9507-ff6e0d466a46', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 227134619, u'size_available': 930343399424, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60989856, u'device': u'/dev/sda1', u'inode_used': 65120, u'block_size': 4096, u'options': u'rw,seclabel,relatime'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227134619, "block_size": 4096, "block_total": 240095108, "block_used": 12960489, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60989856, "inode_total": 61054976, "inode_used": 65120, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930343399424, "size_total": 983429562368, "uuid": "ebbdc914-4a8c-428b-9507-ff6e0d466a46"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:24.681 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:24 +0000 (0:00:00.074) 0:02:02.763 ******* 2020-03-11T22:28:24.718 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2020-03-11T22:28:24.719 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:24.759 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:24 +0000 (0:00:00.077) 0:02:02.840 ******* 2020-03-11T22:28:24.785 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-03-11T22:28:24.787 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:24.822 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:24 +0000 (0:00:00.063) 0:02:02.904 ******* 2020-03-11T22:28:24.859 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-03-11T22:28:24.860 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:24.898 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:24 +0000 (0:00:00.075) 0:02:02.980 ******* 2020-03-11T22:28:24.924 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2020-03-11T22:28:24.925 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:24.962 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:24 +0000 (0:00:00.063) 0:02:03.043 ******* 2020-03-11T22:28:24.998 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2020-03-11T22:28:24.999 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:25.037 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:25 +0000 (0:00:00.074) 0:02:03.118 ******* 2020-03-11T22:28:25.062 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2020-03-11T22:28:25.064 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:25.100 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:25 +0000 (0:00:00.063) 0:02:03.182 ******* 2020-03-11T22:28:25.742 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2020-03-11T22:28:25.743 INFO:teuthology.task.ansible.out:changed: [smithi059.front.sepia.ceph.com] => (item={'value': {u'pvs': u'/dev/nvme0n1'}, 'key': u'vg_nvme'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2020-03-11T22:28:25.781 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:25 +0000 (0:00:00.680) 0:02:03.862 ******* 2020-03-11T22:28:26.331 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-03-11T22:28:26.332 INFO:teuthology.task.ansible.out:changed: [smithi059.front.sepia.ceph.com] => (item={'value': {u'vg': u'vg_nvme', u'size': u'14.9G'}, 'key': u'lv_5'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2020-03-11T22:28:26.671 INFO:teuthology.task.ansible.out:changed: [smithi059.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_4'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-03-11T22:28:27.024 INFO:teuthology.task.ansible.out:changed: [smithi059.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_3'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-03-11T22:28:27.447 INFO:teuthology.task.ansible.out:changed: [smithi059.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_2'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-03-11T22:28:27.792 INFO:teuthology.task.ansible.out:changed: [smithi059.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_1'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-03-11T22:28:27.836 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:27 +0000 (0:00:02.055) 0:02:05.918 ******* 2020-03-11T22:28:28.059 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-03-11T22:28:28.061 INFO:teuthology.task.ansible.out:changed: [smithi059.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2020-03-11T22:28:28.103 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:28 +0000 (0:00:00.266) 0:02:06.184 ******* 2020-03-11T22:28:28.159 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2020-03-11T22:28:28.161 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={'value': {u'vg': u'vg_nvme', u'size': u'14.9G'}, 'key': u'lv_5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:28.367 INFO:teuthology.task.ansible.out:changed: [smithi059.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_4'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-03-11T22:28:28.563 INFO:teuthology.task.ansible.out:changed: [smithi059.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_3'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-03-11T22:28:28.770 INFO:teuthology.task.ansible.out:changed: [smithi059.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_2'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-03-11T22:28:28.967 INFO:teuthology.task.ansible.out:changed: [smithi059.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_1'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-03-11T22:28:29.012 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:29 +0000 (0:00:00.908) 0:02:07.093 ******* 2020-03-11T22:28:29.226 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2020-03-11T22:28:29.228 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2020-03-11T22:28:29.268 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:29 +0000 (0:00:00.256) 0:02:07.350 ******* 2020-03-11T22:28:29.305 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2020-03-11T22:28:29.306 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2020-03-11T22:28:29.346 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:29 +0000 (0:00:00.077) 0:02:07.428 ******* 2020-03-11T22:28:29.833 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2020-03-11T22:28:29.834 INFO:teuthology.task.ansible.out:changed: [smithi059.front.sepia.ceph.com] => {"changed": true} 2020-03-11T22:28:29.883 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:29 +0000 (0:00:00.537) 0:02:07.965 ******* 2020-03-11T22:28:30.355 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2020-03-11T22:28:30.361 INFO:teuthology.task.ansible.out:changed: [smithi059.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2020-03-11T22:28:30.396 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:30 +0000 (0:00:00.512) 0:02:08.478 ******* 2020-03-11T22:28:30.433 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2020-03-11T22:28:30.434 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:30.471 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:30 +0000 (0:00:00.074) 0:02:08.553 ******* 2020-03-11T22:28:30.498 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2020-03-11T22:28:30.500 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:30.541 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:30 +0000 (0:00:00.070) 0:02:08.623 ******* 2020-03-11T22:28:30.567 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2020-03-11T22:28:30.569 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:30.604 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:30 +0000 (0:00:00.062) 0:02:08.685 ******* 2020-03-11T22:28:30.630 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2020-03-11T22:28:30.631 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:30.669 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:30 +0000 (0:00:00.064) 0:02:08.750 ******* 2020-03-11T22:28:30.694 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2020-03-11T22:28:30.696 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:30.731 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:30 +0000 (0:00:00.062) 0:02:08.813 ******* 2020-03-11T22:28:30.760 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2020-03-11T22:28:30.761 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:30.797 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:30 +0000 (0:00:00.065) 0:02:08.879 ******* 2020-03-11T22:28:30.819 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2020-03-11T22:28:30.821 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:30.856 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:30 +0000 (0:00:00.058) 0:02:08.937 ******* 2020-03-11T22:28:30.882 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2020-03-11T22:28:30.883 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:30.919 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:30 +0000 (0:00:00.063) 0:02:09.001 ******* 2020-03-11T22:28:30.945 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2020-03-11T22:28:30.947 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:30.982 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:30 +0000 (0:00:00.062) 0:02:09.063 ******* 2020-03-11T22:28:31.007 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2020-03-11T22:28:31.009 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:31.045 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:31 +0000 (0:00:00.063) 0:02:09.127 ******* 2020-03-11T22:28:31.073 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:31.109 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:31 +0000 (0:00:00.063) 0:02:09.190 ******* 2020-03-11T22:28:31.618 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2020-03-11T22:28:31.620 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => {"changed": false, "checksum": "a7ecef00073feab34f868065d34e9dac55134241", "dest": "/etc/chrony.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/chrony.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 215, "state": "file", "uid": 0} 2020-03-11T22:28:31.665 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:31 +0000 (0:00:00.556) 0:02:09.746 ******* 2020-03-11T22:28:31.998 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2020-03-11T22:28:32.000 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "chronyd", "state": "started", "status": {"ActiveEnterTimestamp": "Wed 2020-03-11 22:25:36 UTC", "ActiveEnterTimestampMonotonic": "16994519", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "tmp.mount basic.target ntpdate.service systemd-tmpfiles-setup.service systemd-journald.socket system.slice -.mount ntpd.service sntp.service sysinit.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Wed 2020-03-11 22:25:35 UTC", "AssertTimestampMonotonic": "15842508", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Wed 2020-03-11 22:25:35 UTC", "ConditionTimestampMonotonic": "15842476", "ConfigurationDirectoryMode": "0755", "Conflicts": "systemd-timesyncd.service ntpd.service shutdown.target", "ControlGroup": "/system.slice/chronyd.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description" 2020-03-11T22:28:32.002 INFO:teuthology.task.ansible.out:: "NTP client/server", "DevicePolicy": "auto", "Documentation": "man:chronyd(8) man:chrony.conf(5)", "DynamicUser": "no", "EnvironmentFiles": "/etc/sysconfig/chronyd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "2345", "ExecMainStartTimestamp": "Wed 2020-03-11 22:25:35 UTC", "ExecMainStartTimestampMonotonic": "15997034", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/chronyd ; argv[]=/usr/sbin/chronyd $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPost": "{ path=/usr/libexec/chrony-helper ; argv[]=/usr/libexec/chrony-helper update-daemon ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/chronyd.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "chronyd.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Wed 2020-03-11 22:25:35 UTC", "InactiveExitTimestampMonotonic": "15863885", "InvocationID": "bb0bbfa0d09d4b6491f9db42de4ce3ba", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "167772 2020-03-11T22:28:32.003 INFO:teuthology.task.ansible.out:16", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "127467", "LimitNPROCSoft": "127467", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127467", "LimitSIGPENDINGSoft": "127467", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "2345", "MemoryAccounting": "yes", "MemoryCurrent": "2809856", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "chronyd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PIDFile": "/run/chrony/chronyd.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "yes", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "full", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target -.mount system.slice", "RequiresMountsFor": "/var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardE 2020-03-11T22:28:32.004 INFO:teuthology.task.ansible.out:rror": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Wed 2020-03-11 22:25:36 UTC", "StateChangeTimestampMonotonic": "16994519", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "26213", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Wed 2020-03-11 22:25:35 UTC", "WatchdogTimestampMonotonic": "15997050", "WatchdogUSec": "0"}} 2020-03-11T22:28:32.047 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:32 +0000 (0:00:00.382) 0:02:10.129 ******* 2020-03-11T22:28:32.074 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2020-03-11T22:28:32.075 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:32.118 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:32 +0000 (0:00:00.070) 0:02:10.199 ******* 2020-03-11T22:28:32.119 INFO:teuthology.task.ansible.out: 2020-03-11T22:28:32.146 INFO:teuthology.task.ansible.out: 2020-03-11T22:28:32.147 INFO:teuthology.task.ansible.out:TASK [testnode : Install python-pip on apt based systems.] ********************* skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:32.190 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:32 +0000 (0:00:00.072) 0:02:10.272 ******* 2020-03-11T22:28:32.216 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2020-03-11T22:28:32.218 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:32.254 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:32 +0000 (0:00:00.064) 0:02:10.336 ******* 2020-03-11T22:28:32.477 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2020-03-11T22:28:32.479 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => {"changed": false, "gid": 1211, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2020-03-11T22:28:32.525 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:32 +0000 (0:00:00.270) 0:02:10.606 ******* 2020-03-11T22:28:33.046 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2020-03-11T22:28:33.047 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1211, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2020-03-11T22:28:33.088 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:33 +0000 (0:00:00.563) 0:02:11.170 ******* 2020-03-11T22:28:34.321 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2020-03-11T22:28:34.322 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages\nRequirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages", "Requirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2020-03-11T22:28:34.363 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:34 +0000 (0:00:01.274) 0:02:12.445 ******* 2020-03-11T22:28:34.389 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2020-03-11T22:28:34.391 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:34.431 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:34 +0000 (0:00:00.067) 0:02:12.512 ******* 2020-03-11T22:28:34.460 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2020-03-11T22:28:34.461 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => {"ansible_facts": {"interface": "ens1f0"}, "changed": false} 2020-03-11T22:28:34.511 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:34 +0000 (0:00:00.080) 0:02:12.593 ******* 2020-03-11T22:28:34.724 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2020-03-11T22:28:34.725 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2020-03-11T22:28:34.763 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:34 +0000 (0:00:00.251) 0:02:12.845 ******* 2020-03-11T22:28:34.790 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2020-03-11T22:28:34.792 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:34.841 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:34 +0000 (0:00:00.078) 0:02:12.923 ******* 2020-03-11T22:28:34.869 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2020-03-11T22:28:34.871 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2020-03-11T22:28:34.907 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:34 +0000 (0:00:00.065) 0:02:12.989 ******* 2020-03-11T22:28:34.933 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2020-03-11T22:28:34.934 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:34.975 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:34 +0000 (0:00:00.067) 0:02:13.056 ******* 2020-03-11T22:28:35.188 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2020-03-11T22:28:35.189 INFO:teuthology.task.ansible.out:changed: [smithi059.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2020-03-11T22:28:35.228 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:35 +0000 (0:00:00.253) 0:02:13.310 ******* 2020-03-11T22:28:35.432 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2020-03-11T22:28:35.433 INFO:teuthology.task.ansible.out:changed: [smithi059.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2020-03-11T22:28:35.488 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:35 +0000 (0:00:00.259) 0:02:13.569 ******* 2020-03-11T22:28:35.515 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2020-03-11T22:28:35.516 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:35.563 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:35 +0000 (0:00:00.075) 0:02:13.645 ******* 2020-03-11T22:28:35.601 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2020-03-11T22:28:35.602 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item=ens1f0) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False"} 2020-03-11T22:28:35.603 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2020-03-11T22:28:35.605 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2020-03-11T22:28:35.616 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2020-03-11T22:28:35.616 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item=ens1f1) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False"} 2020-03-11T22:28:35.659 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:35 +0000 (0:00:00.095) 0:02:13.740 ******* 2020-03-11T22:28:35.695 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2020-03-11T22:28:35.697 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'ens1f0', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi059.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'eno1', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:35.707 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'lo', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:35.708 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'eno2', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:35.720 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'ens1f1', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:35.759 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:35 +0000 (0:00:00.100) 0:02:13.841 ******* 2020-03-11T22:28:35.785 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-03-11T22:28:35.787 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:35.831 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:35 +0000 (0:00:00.071) 0:02:13.913 ******* 2020-03-11T22:28:35.857 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2020-03-11T22:28:35.858 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:35.895 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:35 +0000 (0:00:00.063) 0:02:13.977 ******* 2020-03-11T22:28:35.921 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2020-03-11T22:28:35.922 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:35.960 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:35 +0000 (0:00:00.065) 0:02:14.042 ******* 2020-03-11T22:28:35.987 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2020-03-11T22:28:35.989 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:36.027 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:36 +0000 (0:00:00.066) 0:02:14.108 ******* 2020-03-11T22:28:36.052 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2020-03-11T22:28:36.054 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:36.094 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:36 +0000 (0:00:00.066) 0:02:14.175 ******* 2020-03-11T22:28:36.120 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-03-11T22:28:36.121 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:36.159 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:36 +0000 (0:00:00.064) 0:02:14.240 ******* 2020-03-11T22:28:36.184 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2020-03-11T22:28:36.186 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:36.224 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:36 +0000 (0:00:00.064) 0:02:14.305 ******* 2020-03-11T22:28:36.259 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-03-11T22:28:36.261 INFO:teuthology.task.ansible.out:skipping: [smithi059.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:28:36.304 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:36 +0000 (0:00:00.080) 0:02:14.386 ******* 2020-03-11T22:28:37.567 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2020-03-11T22:28:37.567 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:01.006642", "end": "2020-03-11 22:28:37.511762", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2020-03-11 22:28:36.505120", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:28:37.787 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.003250", "end": "2020-03-11 22:28:37.748324", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2020-03-11 22:28:37.745074", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:28:37.980 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.002946", "end": "2020-03-11 22:28:37.944196", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2020-03-11 22:28:37.941250", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2020-03-11T22:28:38.204 INFO:teuthology.task.ansible.out:ok: [smithi059.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.026647", "end": "2020-03-11 22:28:38.169689", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2020-03-11 22:28:38.143042", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:28:38.259 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:38 +0000 (0:00:01.954) 0:02:16.340 ******* 2020-03-11T22:28:38.285 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2020-03-11T22:28:38.324 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:28:38 +0000 (0:00:00.065) 0:02:16.405 ******* 2020-03-11T22:28:38.527 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2020-03-11T22:28:38.528 INFO:teuthology.task.ansible.out:changed: [smithi059.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2020-03-11T22:28:38.651 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** 2020-03-11T22:28:38.653 INFO:teuthology.task.ansible.out:skipping: no hosts matched 2020-03-11T22:28:38.654 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** 2020-03-11T22:28:38.654 INFO:teuthology.task.ansible.out:skipping: no hosts matched 2020-03-11T22:28:38.656 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** skipping: no hosts matched 2020-03-11T22:28:38.658 INFO:teuthology.task.ansible.out: PLAY RECAP ********************************************************************* smithi059.front.sepia.ceph.com : ok=111 changed=32 unreachable=0 failed=0 skipped=171 rescued=0 ignored=3 Wednesday 11 March 2020 22:28:38 +0000 (0:00:00.333) 0:02:16.738 ******* =============================================================================== 2020-03-11T22:28:38.659 INFO:teuthology.task.ansible.out:common : nrpe - Remove SELinux policy package -------------------------- 17.04s common : nrpe - Load SELinux policy package ---------------------------- 16.95s testnode : Zap all non-root disks --------------------------------------- 7.97s testnode : Install packages --------------------------------------------- 7.83s testnode : Base group workaround for BZ1782899 -------------------------- 5.23s testnode : Ensure ceph dependency packages are not present. ------------- 3.23s testnode : Ensure ceph packages are not present. ------------------------ 2.77s Gathering Facts --------------------------------------------------------- 2.69s testnode : Blow away lingering OSD data and FSIDs ----------------------- 2.07s testnode : Create logical volume(s) ------------------------------------- 2.06s testnode : Rebuild rpmdb ------------------------------------------------ 2.04s testnode : List any leftover Ceph artifacts from previous jobs ---------- 1.95s testnode : Configure local mirrorlists ---------------------------------- 1.66s testnode : Configure version specific repos in /etc/yum.repos.d/ -------- 1.55s common : nrpe - Set SELinux boolean nagios_run_sudo true ---------------- 1.55s common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/. --- 1.54s common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/. --- 1.47s common : Install Kerberos Packages (RedHat) ----------------------------- 1.34s testnode : Install packages via pip ------------------------------------- 1.27s testnode : Set the system's hostname ------------------------------------ 1.22s 2020-03-11T22:28:38.931 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi059.front.sepia.ceph.com')] 2020-03-11T22:28:38.933 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-11T22:28:38.934 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi059.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:28:39.006 INFO:teuthology.orchestra.run.smithi059:> true 2020-03-11T22:28:39.094 INFO:teuthology.run_tasks:Running task clock... 2020-03-11T22:28:39.103 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2020-03-11T22:28:39.104 INFO:teuthology.orchestra.run.smithi059:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2020-03-11T22:28:39.173 INFO:teuthology.orchestra.run.smithi059.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2020-03-11T22:28:39.195 INFO:teuthology.orchestra.run.smithi059.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2020-03-11T22:28:39.244 INFO:teuthology.orchestra.run.smithi059.stderr:sudo: ntpd: command not found 2020-03-11T22:28:39.261 INFO:teuthology.orchestra.run.smithi059.stdout:506 Cannot talk to daemon 2020-03-11T22:28:39.282 INFO:teuthology.orchestra.run.smithi059.stderr:Failed to start ntp.service: Unit ntp.service not found. 2020-03-11T22:28:39.302 INFO:teuthology.orchestra.run.smithi059.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2020-03-11T22:28:39.367 INFO:teuthology.orchestra.run.smithi059.stderr:bash: ntpq: command not found 2020-03-11T22:28:39.370 INFO:teuthology.orchestra.run.smithi059.stdout:210 Number of sources = 4 2020-03-11T22:28:39.370 INFO:teuthology.orchestra.run.smithi059.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2020-03-11T22:28:39.370 INFO:teuthology.orchestra.run.smithi059.stdout:=============================================================================== 2020-03-11T22:28:39.370 INFO:teuthology.orchestra.run.smithi059.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-03-11T22:28:39.370 INFO:teuthology.orchestra.run.smithi059.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-03-11T22:28:39.371 INFO:teuthology.orchestra.run.smithi059.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-03-11T22:28:39.371 INFO:teuthology.orchestra.run.smithi059.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-03-11T22:28:39.371 INFO:teuthology.run_tasks:Running task install... 2020-03-11T22:28:39.380 DEBUG:teuthology.task.install:project ceph 2020-03-11T22:28:39.381 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': '4acc1121e47039a3cdf2978ae458c10d153aa141'}} 2020-03-11T22:28:39.381 DEBUG:teuthology.task.install:config {'sha1': '4acc1121e47039a3cdf2978ae458c10d153aa141'} 2020-03-11T22:28:39.381 INFO:teuthology.task.install:Using flavor: basic 2020-03-11T22:28:39.397 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2020-03-11T22:28:39.397 INFO:teuthology.task.install:extra packages: [] 2020-03-11T22:28:39.398 INFO:teuthology.orchestra.run.smithi059:> sudo lsb_release -is 2020-03-11T22:28:39.451 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS 2020-03-11T22:28:39.452 DEBUG:teuthology.misc:System to be installed: CentOS 2020-03-11T22:28:39.452 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'sha1': '4acc1121e47039a3cdf2978ae458c10d153aa141', 'exclude_packages': [], 'debuginfo': None, 'install_ceph_packages': True, 'wait_for_package': False, 'downgrade_packages': [], 'project': 'ceph', 'tag': None, 'extra_packages': [], 'branch': None, 'packages': {}, 'repos_only': False, 'extra_system_packages': [], 'flavor': 'basic', 'extras': None} 2020-03-11T22:28:39.453 INFO:teuthology.orchestra.run.smithi059:> sudo lsb_release -is 2020-03-11T22:28:39.532 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS 2020-03-11T22:28:39.533 DEBUG:teuthology.misc:System to be installed: CentOS 2020-03-11T22:28:39.534 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=4acc1121e47039a3cdf2978ae458c10d153aa141 2020-03-11T22:28:39.773 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-zafman-octo-diag/4acc1121e47039a3cdf2978ae458c10d153aa141/centos/8/flavors/default/ 2020-03-11T22:28:39.774 INFO:teuthology.task.install.rpm:Package version is 15.1.0-1941.g4acc112 2020-03-11T22:28:39.957 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-zafman-octo-diag/4acc1121e47039a3cdf2978ae458c10d153aa141/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-zafman-octo-diag/4acc1121e47039a3cdf2978ae458c10d153aa141/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-zafman-octo-diag/4acc1121e47039a3cdf2978ae458c10d153aa141/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2020-03-11T22:28:39.958 INFO:teuthology.orchestra.run.smithi059:> sudo sh -c 'cat > /etc/yum.repos.d/ceph.repo' 2020-03-11T22:28:40.002 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2020-03-11T22:28:40.003 INFO:teuthology.orchestra.run.smithi059:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/4acc1121e47039a3cdf2978ae458c10d153aa141/;g' /etc/yum.repos.d/ceph.repo ; fi 2020-03-11T22:28:40.075 INFO:teuthology.orchestra.run.smithi059:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2020-03-11T22:28:40.163 INFO:teuthology.orchestra.run.smithi059:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2020-03-11T22:28:40.193 INFO:teuthology.orchestra.run.smithi059.stdout:check_obsoletes = 1 2020-03-11T22:28:40.195 INFO:teuthology.orchestra.run.smithi059:> sudo yum clean all 2020-03-11T22:28:40.547 INFO:teuthology.orchestra.run.smithi059.stdout:239 files removed 2020-03-11T22:28:40.576 INFO:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-radosgw 2020-03-11T22:28:44.766 INFO:teuthology.orchestra.run.smithi059.stdout:ceph packages for x86_64 18 kB/s | 69 kB 00:03 2020-03-11T22:28:45.139 INFO:teuthology.orchestra.run.smithi059.stdout:ceph noarch packages 40 kB/s | 14 kB 00:00 2020-03-11T22:28:45.470 INFO:teuthology.orchestra.run.smithi059.stdout:ceph source packages 4.6 kB/s | 1.4 kB 00:00 2020-03-11T22:28:45.629 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS-8 - AppStream 47 MB/s | 6.5 MB 00:00 2020-03-11T22:28:46.842 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS-8 - Base 42 MB/s | 5.0 MB 00:00 2020-03-11T22:28:47.673 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS-8 - Extras 32 kB/s | 2.1 kB 00:00 2020-03-11T22:28:47.821 INFO:teuthology.orchestra.run.smithi059.stdout:Copr repo for ceph-el8 owned by ktdreyer 378 kB/s | 38 kB 00:00 2020-03-11T22:28:48.292 INFO:teuthology.orchestra.run.smithi059.stdout:Extra Packages for Enterprise Linux 14 MB/s | 6.0 MB 00:00 2020-03-11T22:28:49.616 INFO:teuthology.orchestra.run.smithi059.stdout:lab-extras 5.6 MB/s | 30 kB 00:00 2020-03-11T22:28:50.616 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2020-03-11T22:28:50.619 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2020-03-11T22:28:50.619 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2020-03-11T22:28:50.619 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2020-03-11T22:28:50.620 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2020-03-11T22:28:50.620 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-radosgw x86_64 2:15.1.0-1941.g4acc112.el8 ceph 9.8 M 2020-03-11T22:28:50.620 INFO:teuthology.orchestra.run.smithi059.stdout:Upgrading: 2020-03-11T22:28:50.620 INFO:teuthology.orchestra.run.smithi059.stdout: librados2 x86_64 2:15.1.0-1941.g4acc112.el8 ceph 3.6 M 2020-03-11T22:28:50.620 INFO:teuthology.orchestra.run.smithi059.stdout: librbd1 x86_64 2:15.1.0-1941.g4acc112.el8 ceph 1.9 M 2020-03-11T22:28:50.620 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2020-03-11T22:28:50.621 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-base x86_64 2:15.1.0-1941.g4acc112.el8 ceph 5.5 M 2020-03-11T22:28:50.621 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-common x86_64 2:15.1.0-1941.g4acc112.el8 ceph 21 M 2020-03-11T22:28:50.621 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-selinux x86_64 2:15.1.0-1941.g4acc112.el8 ceph 25 k 2020-03-11T22:28:50.621 INFO:teuthology.orchestra.run.smithi059.stdout: libcephfs2 x86_64 2:15.1.0-1941.g4acc112.el8 ceph 507 k 2020-03-11T22:28:50.621 INFO:teuthology.orchestra.run.smithi059.stdout: libradosstriper1 x86_64 2:15.1.0-1941.g4acc112.el8 ceph 377 k 2020-03-11T22:28:50.621 INFO:teuthology.orchestra.run.smithi059.stdout: librgw2 x86_64 2:15.1.0-1941.g4acc112.el8 ceph 6.2 M 2020-03-11T22:28:50.621 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ceph-argparse x86_64 2:15.1.0-1941.g4acc112.el8 ceph 40 k 2020-03-11T22:28:50.622 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ceph-common x86_64 2:15.1.0-1941.g4acc112.el8 ceph 49 k 2020-03-11T22:28:50.622 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cephfs x86_64 2:15.1.0-1941.g4acc112.el8 ceph 162 k 2020-03-11T22:28:50.622 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rados x86_64 2:15.1.0-1941.g4acc112.el8 ceph 352 k 2020-03-11T22:28:50.622 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rbd x86_64 2:15.1.0-1941.g4acc112.el8 ceph 309 k 2020-03-11T22:28:50.622 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rgw x86_64 2:15.1.0-1941.g4acc112.el8 ceph 113 k 2020-03-11T22:28:50.622 INFO:teuthology.orchestra.run.smithi059.stdout: librdkafka x86_64 0.11.4-1.el8 CentOS-AppStream 353 k 2020-03-11T22:28:50.622 INFO:teuthology.orchestra.run.smithi059.stdout: lttng-ust x86_64 2.8.1-11.el8 CentOS-AppStream 259 k 2020-03-11T22:28:50.623 INFO:teuthology.orchestra.run.smithi059.stdout: python3-prettytable noarch 0.7.2-14.el8 CentOS-AppStream 44 k 2020-03-11T22:28:50.623 INFO:teuthology.orchestra.run.smithi059.stdout: librabbitmq x86_64 0.9.0-1.el8 CentOS-Base 47 k 2020-03-11T22:28:50.623 INFO:teuthology.orchestra.run.smithi059.stdout: gperftools-libs x86_64 2.7-6.el8 epel 315 k 2020-03-11T22:28:50.623 INFO:teuthology.orchestra.run.smithi059.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2020-03-11T22:28:50.624 INFO:teuthology.orchestra.run.smithi059.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2020-03-11T22:28:50.624 INFO:teuthology.orchestra.run.smithi059.stdout: leveldb x86_64 1.21-2.el8 lab-extras 179 k 2020-03-11T22:28:50.624 INFO:teuthology.orchestra.run.smithi059.stdout: 2020-03-11T22:28:50.624 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2020-03-11T22:28:50.624 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2020-03-11T22:28:50.624 INFO:teuthology.orchestra.run.smithi059.stdout:Install 21 Packages 2020-03-11T22:28:50.624 INFO:teuthology.orchestra.run.smithi059.stdout:Upgrade 2 Packages 2020-03-11T22:28:50.625 INFO:teuthology.orchestra.run.smithi059.stdout: 2020-03-11T22:28:50.625 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 51 M 2020-03-11T22:28:50.625 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2020-03-11T22:28:53.521 INFO:teuthology.orchestra.run.smithi059.stdout:(1/23): ceph-base-15.1.0-1941.g4acc112.el8.x86_ 1.9 MB/s | 5.5 MB 00:02 2020-03-11T22:28:53.590 INFO:teuthology.orchestra.run.smithi059.stdout:(2/23): ceph-selinux-15.1.0-1941.g4acc112.el8.x 361 kB/s | 25 kB 00:00 2020-03-11T22:28:57.528 INFO:teuthology.orchestra.run.smithi059.stdout:(3/23): libcephfs2-15.1.0-1941.g4acc112.el8.x86 129 kB/s | 507 kB 00:03 2020-03-11T22:28:57.821 INFO:teuthology.orchestra.run.smithi059.stdout:(4/23): libradosstriper1-15.1.0-1941.g4acc112.e 1.3 MB/s | 377 kB 00:00 2020-03-11T22:28:58.744 INFO:teuthology.orchestra.run.smithi059.stdout:(5/23): ceph-radosgw-15.1.0-1941.g4acc112.el8.x 1.2 MB/s | 9.8 MB 00:08 2020-03-11T22:28:58.826 INFO:teuthology.orchestra.run.smithi059.stdout:(6/23): python3-ceph-argparse-15.1.0-1941.g4acc 485 kB/s | 40 kB 00:00 2020-03-11T22:28:58.908 INFO:teuthology.orchestra.run.smithi059.stdout:(7/23): python3-ceph-common-15.1.0-1941.g4acc11 599 kB/s | 49 kB 00:00 2020-03-11T22:28:58.997 INFO:teuthology.orchestra.run.smithi059.stdout:(8/23): python3-cephfs-15.1.0-1941.g4acc112.el8 1.8 MB/s | 162 kB 00:00 2020-03-11T22:28:59.096 INFO:teuthology.orchestra.run.smithi059.stdout:(9/23): python3-rados-15.1.0-1941.g4acc112.el8. 3.5 MB/s | 352 kB 00:00 2020-03-11T22:28:59.194 INFO:teuthology.orchestra.run.smithi059.stdout:(10/23): python3-rbd-15.1.0-1941.g4acc112.el8.x 3.1 MB/s | 309 kB 00:00 2020-03-11T22:28:59.283 INFO:teuthology.orchestra.run.smithi059.stdout:(11/23): python3-rgw-15.1.0-1941.g4acc112.el8.x 1.2 MB/s | 113 kB 00:00 2020-03-11T22:28:59.319 INFO:teuthology.orchestra.run.smithi059.stdout:(12/23): librdkafka-0.11.4-1.el8.x86_64.rpm 9.8 MB/s | 353 kB 00:00 2020-03-11T22:28:59.323 INFO:teuthology.orchestra.run.smithi059.stdout:(13/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 63 MB/s | 259 kB 00:00 2020-03-11T22:28:59.325 INFO:teuthology.orchestra.run.smithi059.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 33 MB/s | 44 kB 00:00 2020-03-11T22:28:59.326 INFO:teuthology.orchestra.run.smithi059.stdout:(15/23): librabbitmq-0.9.0-1.el8.x86_64.rpm 29 MB/s | 47 kB 00:00 2020-03-11T22:28:59.420 INFO:teuthology.orchestra.run.smithi059.stdout:(16/23): gperftools-libs-2.7-6.el8.x86_64.rpm 3.3 MB/s | 315 kB 00:00 2020-03-11T22:28:59.437 INFO:teuthology.orchestra.run.smithi059.stdout:(17/23): liboath-2.6.2-3.el8.x86_64.rpm 3.5 MB/s | 59 kB 00:00 2020-03-11T22:28:59.454 INFO:teuthology.orchestra.run.smithi059.stdout:(18/23): libunwind-1.3.1-3.el8.x86_64.rpm 4.4 MB/s | 75 kB 00:00 2020-03-11T22:28:59.460 INFO:teuthology.orchestra.run.smithi059.stdout:(19/23): leveldb-1.21-2.el8.x86_64.rpm 31 MB/s | 179 kB 00:00 2020-03-11T22:28:59.960 INFO:teuthology.orchestra.run.smithi059.stdout:(20/23): librgw2-15.1.0-1941.g4acc112.el8.x86_6 2.9 MB/s | 6.2 MB 00:02 2020-03-11T22:29:00.322 INFO:teuthology.orchestra.run.smithi059.stdout:(21/23): librados2-15.1.0-1941.g4acc112.el8.x86 4.1 MB/s | 3.6 MB 00:00 2020-03-11T22:29:00.451 INFO:teuthology.orchestra.run.smithi059.stdout:(22/23): librbd1-15.1.0-1941.g4acc112.el8.x86_6 3.9 MB/s | 1.9 MB 00:00 2020-03-11T22:29:01.528 INFO:teuthology.orchestra.run.smithi059.stdout:(23/23): ceph-common-15.1.0-1941.g4acc112.el8.x 1.9 MB/s | 21 MB 00:10 2020-03-11T22:29:01.531 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:29:01.531 INFO:teuthology.orchestra.run.smithi059.stdout:Total 4.7 MB/s | 51 MB 00:10 2020-03-11T22:29:01.538 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2020-03-11T22:29:01.590 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2020-03-11T22:29:01.590 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2020-03-11T22:29:01.951 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2020-03-11T22:29:01.951 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2020-03-11T22:29:03.025 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2020-03-11T22:29:03.213 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2020-03-11T22:29:03.375 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2020-03-11T22:29:03.791 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2020-03-11T22:29:03.967 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : librados2-2:15.1.0-1941.g4acc112.el8.x86_64 3/25 2020-03-11T22:29:04.066 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librados2-2:15.1.0-1941.g4acc112.el8.x86_64 3/25 2020-03-11T22:29:04.278 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-rados-2:15.1.0-1941.g4acc112.el8.x86_64 4/25 2020-03-11T22:29:04.450 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libcephfs2-2:15.1.0-1941.g4acc112.el8.x86_64 5/25 2020-03-11T22:29:04.705 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: libcephfs2-2:15.1.0-1941.g4acc112.el8.x86_64 5/25 2020-03-11T22:29:04.925 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : librbd1-2:15.1.0-1941.g4acc112.el8.x86_64 6/25 2020-03-11T22:29:04.998 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librbd1-2:15.1.0-1941.g4acc112.el8.x86_64 6/25 2020-03-11T22:29:05.168 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : librabbitmq-0.9.0-1.el8.x86_64 7/25 2020-03-11T22:29:05.333 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 8/25 2020-03-11T22:29:05.990 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 8/25 2020-03-11T22:29:06.166 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : librgw2-2:15.1.0-1941.g4acc112.el8.x86_64 9/25 2020-03-11T22:29:06.233 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librgw2-2:15.1.0-1941.g4acc112.el8.x86_64 9/25 2020-03-11T22:29:06.393 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-ceph-argparse-2:15.1.0-1941.g4acc112.el8.x 10/25 2020-03-11T22:29:06.545 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cephfs-2:15.1.0-1941.g4acc112.el8.x86_64 11/25 2020-03-11T22:29:06.764 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-rgw-2:15.1.0-1941.g4acc112.el8.x86_64 12/25 2020-03-11T22:29:06.994 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-rbd-2:15.1.0-1941.g4acc112.el8.x86_64 13/25 2020-03-11T22:29:07.133 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libradosstriper1-2:15.1.0-1941.g4acc112.el8.x86_64 14/25 2020-03-11T22:29:07.233 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: libradosstriper1-2:15.1.0-1941.g4acc112.el8.x86_64 14/25 2020-03-11T22:29:07.406 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 15/25 2020-03-11T22:29:07.642 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : gperftools-libs-2.7-6.el8.x86_64 16/25 2020-03-11T22:29:07.825 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : leveldb-1.21-2.el8.x86_64 17/25 2020-03-11T22:29:07.984 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2020-03-11T22:29:08.150 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-ceph-common-2:15.1.0-1941.g4acc112.el8.x86 19/25 2020-03-11T22:29:11.972 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-common-2:15.1.0-1941.g4acc112.el8.x86_64 20/25 2020-03-11T22:29:12.166 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-common-2:15.1.0-1941.g4acc112.el8.x86_64 20/25 2020-03-11T22:29:12.878 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-common-2:15.1.0-1941.g4acc112.el8.x86_64 20/25 2020-03-11T22:29:13.176 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-base-2:15.1.0-1941.g4acc112.el8.x86_64 21/25 2020-03-11T22:29:13.264 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-base-2:15.1.0-1941.g4acc112.el8.x86_64 21/25 2020-03-11T22:29:13.374 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-selinux-2:15.1.0-1941.g4acc112.el8.x86_64 22/25 2020-03-11T22:29:30.288 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-selinux-2:15.1.0-1941.g4acc112.el8.x86_64 22/25 2020-03-11T22:29:30.288 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /sys 2020-03-11T22:29:30.288 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /proc 2020-03-11T22:29:30.289 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /dev 2020-03-11T22:29:30.289 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /run 2020-03-11T22:29:30.289 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /mnt 2020-03-11T22:29:30.289 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /var/tmp 2020-03-11T22:29:30.290 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /home 2020-03-11T22:29:30.290 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /tmp 2020-03-11T22:29:30.290 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /dev 2020-03-11T22:29:30.290 INFO:teuthology.orchestra.run.smithi059.stdout: 2020-03-11T22:29:31.405 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-radosgw-2:15.1.0-1941.g4acc112.el8.x86_64 23/25 2020-03-11T22:29:31.482 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-radosgw-2:15.1.0-1941.g4acc112.el8.x86_64 23/25 2020-03-11T22:29:31.483 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2020-03-11T22:29:31.723 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2020-03-11T22:29:31.724 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2020-03-11T22:29:32.407 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2020-03-11T22:29:32.407 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-base-2:15.1.0-1941.g4acc112.el8.x86_64 1/25 2020-03-11T22:29:32.407 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-common-2:15.1.0-1941.g4acc112.el8.x86_64 2/25 2020-03-11T22:29:32.407 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-radosgw-2:15.1.0-1941.g4acc112.el8.x86_64 3/25 2020-03-11T22:29:32.408 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-selinux-2:15.1.0-1941.g4acc112.el8.x86_64 4/25 2020-03-11T22:29:32.408 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libcephfs2-2:15.1.0-1941.g4acc112.el8.x86_64 5/25 2020-03-11T22:29:32.408 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libradosstriper1-2:15.1.0-1941.g4acc112.el8.x86_64 6/25 2020-03-11T22:29:32.408 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librgw2-2:15.1.0-1941.g4acc112.el8.x86_64 7/25 2020-03-11T22:29:32.408 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-ceph-argparse-2:15.1.0-1941.g4acc112.el8.x 8/25 2020-03-11T22:29:32.408 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-ceph-common-2:15.1.0-1941.g4acc112.el8.x86 9/25 2020-03-11T22:29:32.408 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cephfs-2:15.1.0-1941.g4acc112.el8.x86_64 10/25 2020-03-11T22:29:32.409 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-rados-2:15.1.0-1941.g4acc112.el8.x86_64 11/25 2020-03-11T22:29:32.409 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-rbd-2:15.1.0-1941.g4acc112.el8.x86_64 12/25 2020-03-11T22:29:32.409 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-rgw-2:15.1.0-1941.g4acc112.el8.x86_64 13/25 2020-03-11T22:29:32.409 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 14/25 2020-03-11T22:29:32.409 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2020-03-11T22:29:32.409 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2020-03-11T22:29:32.409 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librabbitmq-0.9.0-1.el8.x86_64 17/25 2020-03-11T22:29:32.410 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : gperftools-libs-2.7-6.el8.x86_64 18/25 2020-03-11T22:29:32.410 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2020-03-11T22:29:32.410 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2020-03-11T22:29:32.410 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : leveldb-1.21-2.el8.x86_64 21/25 2020-03-11T22:29:32.410 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librados2-2:15.1.0-1941.g4acc112.el8.x86_64 22/25 2020-03-11T22:29:32.410 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2020-03-11T22:29:32.411 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librbd1-2:15.1.0-1941.g4acc112.el8.x86_64 24/25 2020-03-11T22:29:33.244 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2020-03-11T22:29:33.244 INFO:teuthology.orchestra.run.smithi059.stdout: 2020-03-11T22:29:33.244 INFO:teuthology.orchestra.run.smithi059.stdout:Upgraded: 2020-03-11T22:29:33.245 INFO:teuthology.orchestra.run.smithi059.stdout: librados2-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:29:33.245 INFO:teuthology.orchestra.run.smithi059.stdout: librbd1-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:29:33.245 INFO:teuthology.orchestra.run.smithi059.stdout: 2020-03-11T22:29:33.245 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2020-03-11T22:29:33.245 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-radosgw-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:29:33.245 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-base-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:29:33.246 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-common-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:29:33.246 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-selinux-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:29:33.246 INFO:teuthology.orchestra.run.smithi059.stdout: libcephfs2-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:29:33.246 INFO:teuthology.orchestra.run.smithi059.stdout: libradosstriper1-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:29:33.246 INFO:teuthology.orchestra.run.smithi059.stdout: librgw2-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:29:33.246 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ceph-argparse-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:29:33.247 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ceph-common-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:29:33.247 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cephfs-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:29:33.247 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rados-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:29:33.247 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rbd-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:29:33.247 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rgw-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:29:33.247 INFO:teuthology.orchestra.run.smithi059.stdout: librdkafka-0.11.4-1.el8.x86_64 2020-03-11T22:29:33.247 INFO:teuthology.orchestra.run.smithi059.stdout: lttng-ust-2.8.1-11.el8.x86_64 2020-03-11T22:29:33.248 INFO:teuthology.orchestra.run.smithi059.stdout: python3-prettytable-0.7.2-14.el8.noarch 2020-03-11T22:29:33.248 INFO:teuthology.orchestra.run.smithi059.stdout: librabbitmq-0.9.0-1.el8.x86_64 2020-03-11T22:29:33.248 INFO:teuthology.orchestra.run.smithi059.stdout: gperftools-libs-2.7-6.el8.x86_64 2020-03-11T22:29:33.248 INFO:teuthology.orchestra.run.smithi059.stdout: liboath-2.6.2-3.el8.x86_64 2020-03-11T22:29:33.248 INFO:teuthology.orchestra.run.smithi059.stdout: libunwind-1.3.1-3.el8.x86_64 2020-03-11T22:29:33.248 INFO:teuthology.orchestra.run.smithi059.stdout: leveldb-1.21-2.el8.x86_64 2020-03-11T22:29:33.249 INFO:teuthology.orchestra.run.smithi059.stdout: 2020-03-11T22:29:33.249 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2020-03-11T22:29:33.474 INFO:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-test 2020-03-11T22:29:33.907 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:00:44 ago on Wed 11 Mar 2020 10:28:49 PM UTC. 2020-03-11T22:29:34.294 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2020-03-11T22:29:34.296 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2020-03-11T22:29:34.296 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2020-03-11T22:29:34.296 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2020-03-11T22:29:34.296 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2020-03-11T22:29:34.296 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-test x86_64 2:15.1.0-1941.g4acc112.el8 ceph 42 M 2020-03-11T22:29:34.296 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2020-03-11T22:29:34.297 INFO:teuthology.orchestra.run.smithi059.stdout: jq x86_64 1.5-12.el8 CentOS-AppStream 161 k 2020-03-11T22:29:34.297 INFO:teuthology.orchestra.run.smithi059.stdout: oniguruma x86_64 6.8.2-1.el8 CentOS-AppStream 188 k 2020-03-11T22:29:34.297 INFO:teuthology.orchestra.run.smithi059.stdout: socat x86_64 1.7.3.2-6.el8 CentOS-AppStream 298 k 2020-03-11T22:29:34.297 INFO:teuthology.orchestra.run.smithi059.stdout: libxslt x86_64 1.1.32-3.el8 CentOS-Base 249 k 2020-03-11T22:29:34.297 INFO:teuthology.orchestra.run.smithi059.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2020-03-11T22:29:34.297 INFO:teuthology.orchestra.run.smithi059.stdout: 2020-03-11T22:29:34.297 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2020-03-11T22:29:34.298 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2020-03-11T22:29:34.298 INFO:teuthology.orchestra.run.smithi059.stdout:Install 6 Packages 2020-03-11T22:29:34.298 INFO:teuthology.orchestra.run.smithi059.stdout: 2020-03-11T22:29:34.298 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 43 M 2020-03-11T22:29:34.299 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 184 M 2020-03-11T22:29:34.299 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2020-03-11T22:29:34.337 INFO:teuthology.orchestra.run.smithi059.stdout:(1/6): jq-1.5-12.el8.x86_64.rpm 4.1 MB/s | 161 kB 00:00 2020-03-11T22:29:34.342 INFO:teuthology.orchestra.run.smithi059.stdout:(2/6): oniguruma-6.8.2-1.el8.x86_64.rpm 4.2 MB/s | 188 kB 00:00 2020-03-11T22:29:34.344 INFO:teuthology.orchestra.run.smithi059.stdout:(3/6): socat-1.7.3.2-6.el8.x86_64.rpm 42 MB/s | 298 kB 00:00 2020-03-11T22:29:34.347 INFO:teuthology.orchestra.run.smithi059.stdout:(4/6): libxslt-1.1.32-3.el8.x86_64.rpm 47 MB/s | 249 kB 00:00 2020-03-11T22:29:34.406 INFO:teuthology.orchestra.run.smithi059.stdout:(5/6): xmlstarlet-1.6.1-11.el8.x86_64.rpm 1.1 MB/s | 69 kB 00:00 2020-03-11T22:29:36.573 INFO:teuthology.orchestra.run.smithi059.stdout:(6/6): ceph-test-15.1.0-1941.g4acc112.el8.x86_6 18 MB/s | 42 MB 00:02 2020-03-11T22:29:36.577 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:29:36.577 INFO:teuthology.orchestra.run.smithi059.stdout:Total 19 MB/s | 43 MB 00:02 2020-03-11T22:29:36.583 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2020-03-11T22:29:36.608 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2020-03-11T22:29:36.608 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2020-03-11T22:29:36.879 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2020-03-11T22:29:36.880 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2020-03-11T22:29:37.344 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2020-03-11T22:29:37.534 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libxslt-1.1.32-3.el8.x86_64 1/6 2020-03-11T22:29:37.717 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 2/6 2020-03-11T22:29:38.619 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : socat-1.7.3.2-6.el8.x86_64 3/6 2020-03-11T22:29:38.779 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : oniguruma-6.8.2-1.el8.x86_64 4/6 2020-03-11T22:29:38.863 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: oniguruma-6.8.2-1.el8.x86_64 4/6 2020-03-11T22:29:43.201 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : jq-1.5-12.el8.x86_64 5/6 2020-03-11T22:29:43.400 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-test-2:15.1.0-1941.g4acc112.el8.x86_64 6/6 2020-03-11T22:29:43.639 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-test-2:15.1.0-1941.g4acc112.el8.x86_64 6/6 2020-03-11T22:29:43.639 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-test-2:15.1.0-1941.g4acc112.el8.x86_64 1/6 2020-03-11T22:29:43.639 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : jq-1.5-12.el8.x86_64 2/6 2020-03-11T22:29:43.639 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : oniguruma-6.8.2-1.el8.x86_64 3/6 2020-03-11T22:29:43.640 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : socat-1.7.3.2-6.el8.x86_64 4/6 2020-03-11T22:29:43.640 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libxslt-1.1.32-3.el8.x86_64 5/6 2020-03-11T22:29:44.642 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 6/6 2020-03-11T22:29:44.643 INFO:teuthology.orchestra.run.smithi059.stdout: 2020-03-11T22:29:44.643 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2020-03-11T22:29:44.643 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-test-2:15.1.0-1941.g4acc112.el8.x86_64 jq-1.5-12.el8.x86_64 2020-03-11T22:29:44.643 INFO:teuthology.orchestra.run.smithi059.stdout: oniguruma-6.8.2-1.el8.x86_64 socat-1.7.3.2-6.el8.x86_64 2020-03-11T22:29:44.643 INFO:teuthology.orchestra.run.smithi059.stdout: libxslt-1.1.32-3.el8.x86_64 xmlstarlet-1.6.1-11.el8.x86_64 2020-03-11T22:29:44.643 INFO:teuthology.orchestra.run.smithi059.stdout: 2020-03-11T22:29:44.644 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2020-03-11T22:29:45.733 INFO:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph 2020-03-11T22:29:46.169 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:00:57 ago on Wed 11 Mar 2020 10:28:49 PM UTC. 2020-03-11T22:29:46.589 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2020-03-11T22:29:46.593 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2020-03-11T22:29:46.593 INFO:teuthology.orchestra.run.smithi059.stdout: Package ArchVersion Repository Size 2020-03-11T22:29:46.593 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2020-03-11T22:29:46.594 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2020-03-11T22:29:46.594 INFO:teuthology.orchestra.run.smithi059.stdout: ceph x86_642:15.1.0-1941.g4acc112.el8 ceph 6.4 k 2020-03-11T22:29:46.594 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2020-03-11T22:29:46.594 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mds x86_642:15.1.0-1941.g4acc112.el8 ceph 2.0 M 2020-03-11T22:29:46.594 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr x86_642:15.1.0-1941.g4acc112.el8 ceph 1.4 M 2020-03-11T22:29:46.595 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mon x86_642:15.1.0-1941.g4acc112.el8 ceph 4.4 M 2020-03-11T22:29:46.595 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-osd x86_642:15.1.0-1941.g4acc112.el8 ceph 17 M 2020-03-11T22:29:46.595 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-modules-core noarch2:15.1.0-1941.g4acc112.el8 ceph-noarch 165 k 2020-03-11T22:29:46.595 INFO:teuthology.orchestra.run.smithi059.stdout: python3-lxml x86_644.2.3-1.el8 CentOS-AppStream 1.5 M 2020-03-11T22:29:46.596 INFO:teuthology.orchestra.run.smithi059.stdout: python3-mako noarch1.0.6-13.el8 CentOS-AppStream 157 k 2020-03-11T22:29:46.596 INFO:teuthology.orchestra.run.smithi059.stdout: python3-markupsafe x86_640.23-19.el8 CentOS-AppStream 39 k 2020-03-11T22:29:46.596 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pytz noarch2017.2-9.el8 CentOS-AppStream 54 k 2020-03-11T22:29:46.596 INFO:teuthology.orchestra.run.smithi059.stdout: python3-webencodings noarch0.5.1-6.el8 CentOS-AppStream 27 k 2020-03-11T22:29:46.597 INFO:teuthology.orchestra.run.smithi059.stdout: python3-werkzeug noarch0.12.2-4.el8 CentOS-AppStream 457 k 2020-03-11T22:29:46.597 INFO:teuthology.orchestra.run.smithi059.stdout: python3-chardet noarch3.0.4-7.el8 CentOS-Base 195 k 2020-03-11T22:29:46.597 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pysocks noarch1.6.8-3.el8 CentOS-Base 34 k 2020-03-11T22:29:46.597 INFO:teuthology.orchestra.run.smithi059.stdout: python3-requests noarch2.20.0-2.1.el8_1 CentOS-Base 123 k 2020-03-11T22:29:46.598 INFO:teuthology.orchestra.run.smithi059.stdout: python3-urllib3 noarch1.24.2-2.el8 CentOS-Base 176 k 2020-03-11T22:29:46.598 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cheroot noarch8.2.1-1.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 145 k 2020-03-11T22:29:46.598 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cherrypy noarch18.4.0-1.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 385 k 2020-03-11T22:29:46.598 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco noarch6.2-6.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 11 k 2020-03-11T22:29:46.599 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-functools noarch2.0-4.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 18 k 2020-03-11T22:29:46.599 INFO:teuthology.orchestra.run.smithi059.stdout: python3-logutils noarch0.3.5-11.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 49 k 2020-03-11T22:29:46.599 INFO:teuthology.orchestra.run.smithi059.stdout: python3-more-itertools noarch7.2.0-3.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 59 k 2020-03-11T22:29:46.599 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pecan noarch1.3.2-9.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 283 k 2020-03-11T22:29:46.600 INFO:teuthology.orchestra.run.smithi059.stdout: python3-portend noarch2.6-1.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 16 k 2020-03-11T22:29:46.600 INFO:teuthology.orchestra.run.smithi059.stdout: python3-simplegeneric noarch0.8.1-17.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 19 k 2020-03-11T22:29:46.600 INFO:teuthology.orchestra.run.smithi059.stdout: python3-singledispatch noarch3.4.0.3-18.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 24 k 2020-03-11T22:29:46.600 INFO:teuthology.orchestra.run.smithi059.stdout: python3-tempora noarch1.14.1-5.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 30 k 2020-03-11T22:29:46.600 INFO:teuthology.orchestra.run.smithi059.stdout: python3-trustme noarch0.5.2-4.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 27 k 2020-03-11T22:29:46.601 INFO:teuthology.orchestra.run.smithi059.stdout: python3-zc-lockfile noarch2.0-2.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 23 k 2020-03-11T22:29:46.601 INFO:teuthology.orchestra.run.smithi059.stdout: python3-bcrypt x86_643.1.6-2.el8.1 epel 44 k 2020-03-11T22:29:46.601 INFO:teuthology.orchestra.run.smithi059.stdout: python3-beautifulsoup4 noarch4.6.3-2.el8.1 epel 185 k 2020-03-11T22:29:46.601 INFO:teuthology.orchestra.run.smithi059.stdout: python3-waitress noarch1.2.1-2.el8.1 epel 241 k 2020-03-11T22:29:46.602 INFO:teuthology.orchestra.run.smithi059.stdout: python3-webob noarch1.8.5-1.el8.1 epel 251 k 2020-03-11T22:29:46.602 INFO:teuthology.orchestra.run.smithi059.stdout: python3-webtest noarch2.0.33-1.el8 epel 85 k 2020-03-11T22:29:46.602 INFO:teuthology.orchestra.run.smithi059.stdout:Installing weak dependencies: 2020-03-11T22:29:46.602 INFO:teuthology.orchestra.run.smithi059.stdout: python3-html5lib noarch1:0.999999999-6.el8 CentOS-AppStream 214 k 2020-03-11T22:29:46.603 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cssselect noarch0.9.2-10.el8 epel 40 k 2020-03-11T22:29:46.603 INFO:teuthology.orchestra.run.smithi059.stdout: 2020-03-11T22:29:46.603 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2020-03-11T22:29:46.603 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2020-03-11T22:29:46.603 INFO:teuthology.orchestra.run.smithi059.stdout:Install 36 Packages 2020-03-11T22:29:46.604 INFO:teuthology.orchestra.run.smithi059.stdout: 2020-03-11T22:29:46.605 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 29 M 2020-03-11T22:29:46.605 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 105 M 2020-03-11T22:29:46.605 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2020-03-11T22:29:53.691 INFO:teuthology.orchestra.run.smithi059.stdout:(1/36): ceph-15.1.0-1941.g4acc112.el8.x86_64.rp 928 B/s | 6.4 kB 00:07 2020-03-11T22:29:53.988 INFO:teuthology.orchestra.run.smithi059.stdout:(2/36): ceph-mgr-15.1.0-1941.g4acc112.el8.x86_6 196 kB/s | 1.4 MB 00:07 2020-03-11T22:29:54.006 INFO:teuthology.orchestra.run.smithi059.stdout:(3/36): ceph-mds-15.1.0-1941.g4acc112.el8.x86_6 281 kB/s | 2.0 MB 00:07 2020-03-11T22:29:54.087 INFO:teuthology.orchestra.run.smithi059.stdout:(4/36): ceph-mgr-modules-core-15.1.0-1941.g4acc 2.0 MB/s | 165 kB 00:00 2020-03-11T22:29:54.120 INFO:teuthology.orchestra.run.smithi059.stdout:(5/36): python3-html5lib-0.999999999-6.el8.noar 6.4 MB/s | 214 kB 00:00 2020-03-11T22:29:54.144 INFO:teuthology.orchestra.run.smithi059.stdout:(6/36): python3-lxml-4.2.3-1.el8.x86_64.rpm 62 MB/s | 1.5 MB 00:00 2020-03-11T22:29:54.149 INFO:teuthology.orchestra.run.smithi059.stdout:(7/36): python3-mako-1.0.6-13.el8.noarch.rpm 33 MB/s | 157 kB 00:00 2020-03-11T22:29:54.151 INFO:teuthology.orchestra.run.smithi059.stdout:(8/36): python3-markupsafe-0.23-19.el8.x86_64.r 21 MB/s | 39 kB 00:00 2020-03-11T22:29:54.153 INFO:teuthology.orchestra.run.smithi059.stdout:(9/36): python3-pytz-2017.2-9.el8.noarch.rpm 24 MB/s | 54 kB 00:00 2020-03-11T22:29:54.156 INFO:teuthology.orchestra.run.smithi059.stdout:(10/36): python3-webencodings-0.5.1-6.el8.noarc 14 MB/s | 27 kB 00:00 2020-03-11T22:29:54.170 INFO:teuthology.orchestra.run.smithi059.stdout:(11/36): ceph-mon-15.1.0-1941.g4acc112.el8.x86_ 9.1 MB/s | 4.4 MB 00:00 2020-03-11T22:29:54.187 INFO:teuthology.orchestra.run.smithi059.stdout:(12/36): python3-werkzeug-0.12.2-4.el8.noarch.r 14 MB/s | 457 kB 00:00 2020-03-11T22:29:54.188 INFO:teuthology.orchestra.run.smithi059.stdout:(13/36): python3-pysocks-1.6.8-3.el8.noarch.rpm 26 MB/s | 34 kB 00:00 2020-03-11T22:29:54.191 INFO:teuthology.orchestra.run.smithi059.stdout:(14/36): python3-requests-2.20.0-2.1.el8_1.noar 53 MB/s | 123 kB 00:00 2020-03-11T22:29:54.194 INFO:teuthology.orchestra.run.smithi059.stdout:(15/36): python3-urllib3-1.24.2-2.el8.noarch.rp 59 MB/s | 176 kB 00:00 2020-03-11T22:29:54.209 INFO:teuthology.orchestra.run.smithi059.stdout:(16/36): python3-chardet-3.0.4-7.el8.noarch.rpm 4.9 MB/s | 195 kB 00:00 2020-03-11T22:29:54.249 INFO:teuthology.orchestra.run.smithi059.stdout:(17/36): python3-cheroot-8.2.1-1.el8.noarch.rpm 2.6 MB/s | 145 kB 00:00 2020-03-11T22:29:54.257 INFO:teuthology.orchestra.run.smithi059.stdout:(18/36): python3-jaraco-6.2-6.el8.noarch.rpm 1.4 MB/s | 11 kB 00:00 2020-03-11T22:29:54.266 INFO:teuthology.orchestra.run.smithi059.stdout:(19/36): python3-jaraco-functools-2.0-4.el8.noa 2.0 MB/s | 18 kB 00:00 2020-03-11T22:29:54.274 INFO:teuthology.orchestra.run.smithi059.stdout:(20/36): python3-cherrypy-18.4.0-1.el8.noarch.r 5.8 MB/s | 385 kB 00:00 2020-03-11T22:29:54.276 INFO:teuthology.orchestra.run.smithi059.stdout:(21/36): python3-logutils-0.3.5-11.el8.noarch.r 5.1 MB/s | 49 kB 00:00 2020-03-11T22:29:54.282 INFO:teuthology.orchestra.run.smithi059.stdout:(22/36): python3-more-itertools-7.2.0-3.el8.noa 6.7 MB/s | 59 kB 00:00 2020-03-11T22:29:54.291 INFO:teuthology.orchestra.run.smithi059.stdout:(23/36): python3-portend-2.6-1.el8.noarch.rpm 1.9 MB/s | 16 kB 00:00 2020-03-11T22:29:54.300 INFO:teuthology.orchestra.run.smithi059.stdout:(24/36): python3-pecan-1.3.2-9.el8.noarch.rpm 12 MB/s | 283 kB 00:00 2020-03-11T22:29:54.301 INFO:teuthology.orchestra.run.smithi059.stdout:(25/36): python3-simplegeneric-0.8.1-17.el8.noa 1.8 MB/s | 19 kB 00:00 2020-03-11T22:29:54.308 INFO:teuthology.orchestra.run.smithi059.stdout:(26/36): python3-singledispatch-3.4.0.3-18.el8. 2.8 MB/s | 24 kB 00:00 2020-03-11T22:29:54.310 INFO:teuthology.orchestra.run.smithi059.stdout:(27/36): python3-tempora-1.14.1-5.el8.noarch.rp 3.4 MB/s | 30 kB 00:00 2020-03-11T22:29:54.318 INFO:teuthology.orchestra.run.smithi059.stdout:(28/36): python3-trustme-0.5.2-4.el8.noarch.rpm 2.7 MB/s | 27 kB 00:00 2020-03-11T22:29:54.319 INFO:teuthology.orchestra.run.smithi059.stdout:(29/36): python3-zc-lockfile-2.0-2.el8.noarch.r 2.6 MB/s | 23 kB 00:00 2020-03-11T22:29:54.380 INFO:teuthology.orchestra.run.smithi059.stdout:(30/36): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 706 kB/s | 44 kB 00:00 2020-03-11T22:29:54.411 INFO:teuthology.orchestra.run.smithi059.stdout:(31/36): python3-cssselect-0.9.2-10.el8.noarch. 1.3 MB/s | 40 kB 00:00 2020-03-11T22:29:54.412 INFO:teuthology.orchestra.run.smithi059.stdout:(32/36): python3-beautifulsoup4-4.6.3-2.el8.1.n 1.9 MB/s | 185 kB 00:00 2020-03-11T22:29:54.450 INFO:teuthology.orchestra.run.smithi059.stdout:(33/36): python3-webob-1.8.5-1.el8.1.noarch.rpm 6.5 MB/s | 251 kB 00:00 2020-03-11T22:29:54.458 INFO:teuthology.orchestra.run.smithi059.stdout:(34/36): python3-waitress-1.2.1-2.el8.1.noarch. 5.0 MB/s | 241 kB 00:00 2020-03-11T22:29:54.470 INFO:teuthology.orchestra.run.smithi059.stdout:(35/36): python3-webtest-2.0.33-1.el8.noarch.rp 4.2 MB/s | 85 kB 00:00 2020-03-11T22:29:54.780 INFO:teuthology.orchestra.run.smithi059.stdout:(36/36): ceph-osd-15.1.0-1941.g4acc112.el8.x86_ 21 MB/s | 17 MB 00:00 2020-03-11T22:29:54.783 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:29:54.784 INFO:teuthology.orchestra.run.smithi059.stdout:Total 3.6 MB/s | 29 MB 00:08 2020-03-11T22:29:54.807 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2020-03-11T22:29:54.848 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2020-03-11T22:29:54.849 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2020-03-11T22:29:55.176 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2020-03-11T22:29:55.177 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2020-03-11T22:29:56.525 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2020-03-11T22:29:56.730 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 1/36 2020-03-11T22:29:56.961 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 2/36 2020-03-11T22:29:57.170 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 3/36 2020-03-11T22:29:57.311 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cssselect-0.9.2-10.el8.noarch 4/36 2020-03-11T22:29:57.485 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 5/36 2020-03-11T22:29:57.668 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 6/36 2020-03-11T22:29:57.817 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-trustme-0.5.2-4.el8.noarch 7/36 2020-03-11T22:29:57.990 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 8/36 2020-03-11T22:29:58.140 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 9/36 2020-03-11T22:29:58.321 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 10/36 2020-03-11T22:29:58.466 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 11/36 2020-03-11T22:29:58.696 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 12/36 2020-03-11T22:29:58.943 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cheroot-8.2.1-1.el8.noarch 13/36 2020-03-11T22:29:59.161 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pysocks-1.6.8-3.el8.noarch 14/36 2020-03-11T22:29:59.435 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-urllib3-1.24.2-2.el8.noarch 15/36 2020-03-11T22:29:59.619 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-chardet-3.0.4-7.el8.noarch 16/36 2020-03-11T22:29:59.862 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-requests-2.20.0-2.1.el8_1.noarch 17/36 2020-03-11T22:30:00.063 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 18/36 2020-03-11T22:30:00.307 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-webencodings-0.5.1-6.el8.noarch 19/36 2020-03-11T22:30:00.550 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-html5lib-1:0.999999999-6.el8.noarch 20/36 2020-03-11T22:30:00.887 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 21/36 2020-03-11T22:30:01.145 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-lxml-4.2.3-1.el8.x86_64 22/36 2020-03-11T22:30:01.472 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 23/36 2020-03-11T22:30:02.123 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 24/36 2020-03-11T22:30:02.257 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 25/36 2020-03-11T22:30:02.520 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-portend-2.6-1.el8.noarch 26/36 2020-03-11T22:30:02.745 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 27/36 2020-03-11T22:30:02.960 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 28/36 2020-03-11T22:30:03.289 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 29/36 2020-03-11T22:30:03.578 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 30/36 2020-03-11T22:30:03.951 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-modules-core-2:15.1.0-1941.g4acc112.el8.n 31/36 2020-03-11T22:30:04.126 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-2:15.1.0-1941.g4acc112.el8.x86_64 32/36 2020-03-11T22:30:05.631 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-2:15.1.0-1941.g4acc112.el8.x86_64 32/36 2020-03-11T22:30:05.817 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-osd-2:15.1.0-1941.g4acc112.el8.x86_64 33/36 2020-03-11T22:30:06.230 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-osd-2:15.1.0-1941.g4acc112.el8.x86_64 33/36 2020-03-11T22:30:06.367 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mon-2:15.1.0-1941.g4acc112.el8.x86_64 34/36 2020-03-11T22:30:06.574 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mon-2:15.1.0-1941.g4acc112.el8.x86_64 34/36 2020-03-11T22:30:06.717 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mds-2:15.1.0-1941.g4acc112.el8.x86_64 35/36 2020-03-11T22:30:06.737 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mds-2:15.1.0-1941.g4acc112.el8.x86_64 35/36 2020-03-11T22:30:06.832 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-2:15.1.0-1941.g4acc112.el8.x86_64 36/36 2020-03-11T22:30:07.188 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-2:15.1.0-1941.g4acc112.el8.x86_64 36/36 2020-03-11T22:30:07.189 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-2:15.1.0-1941.g4acc112.el8.x86_64 1/36 2020-03-11T22:30:07.189 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mds-2:15.1.0-1941.g4acc112.el8.x86_64 2/36 2020-03-11T22:30:07.189 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-2:15.1.0-1941.g4acc112.el8.x86_64 3/36 2020-03-11T22:30:07.189 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mon-2:15.1.0-1941.g4acc112.el8.x86_64 4/36 2020-03-11T22:30:07.190 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-osd-2:15.1.0-1941.g4acc112.el8.x86_64 5/36 2020-03-11T22:30:07.190 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-modules-core-2:15.1.0-1941.g4acc112.el8.n 6/36 2020-03-11T22:30:07.190 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-html5lib-1:0.999999999-6.el8.noarch 7/36 2020-03-11T22:30:07.190 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-lxml-4.2.3-1.el8.x86_64 8/36 2020-03-11T22:30:07.191 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 9/36 2020-03-11T22:30:07.191 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 10/36 2020-03-11T22:30:07.191 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 11/36 2020-03-11T22:30:07.191 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-webencodings-0.5.1-6.el8.noarch 12/36 2020-03-11T22:30:07.192 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 13/36 2020-03-11T22:30:07.192 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-chardet-3.0.4-7.el8.noarch 14/36 2020-03-11T22:30:07.192 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pysocks-1.6.8-3.el8.noarch 15/36 2020-03-11T22:30:07.192 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-requests-2.20.0-2.1.el8_1.noarch 16/36 2020-03-11T22:30:07.192 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-urllib3-1.24.2-2.el8.noarch 17/36 2020-03-11T22:30:07.193 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cheroot-8.2.1-1.el8.noarch 18/36 2020-03-11T22:30:07.193 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 19/36 2020-03-11T22:30:07.193 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 20/36 2020-03-11T22:30:07.193 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 21/36 2020-03-11T22:30:07.194 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 22/36 2020-03-11T22:30:07.194 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 23/36 2020-03-11T22:30:07.194 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 24/36 2020-03-11T22:30:07.194 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-portend-2.6-1.el8.noarch 25/36 2020-03-11T22:30:07.195 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 26/36 2020-03-11T22:30:07.195 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 27/36 2020-03-11T22:30:07.195 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 28/36 2020-03-11T22:30:07.195 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-trustme-0.5.2-4.el8.noarch 29/36 2020-03-11T22:30:07.195 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 30/36 2020-03-11T22:30:07.196 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 31/36 2020-03-11T22:30:07.196 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 32/36 2020-03-11T22:30:07.196 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cssselect-0.9.2-10.el8.noarch 33/36 2020-03-11T22:30:07.196 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 34/36 2020-03-11T22:30:07.197 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 35/36 2020-03-11T22:30:07.836 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 36/36 2020-03-11T22:30:07.837 INFO:teuthology.orchestra.run.smithi059.stdout: 2020-03-11T22:30:07.837 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2020-03-11T22:30:07.837 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:30:07.837 INFO:teuthology.orchestra.run.smithi059.stdout: python3-html5lib-1:0.999999999-6.el8.noarch 2020-03-11T22:30:07.837 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cssselect-0.9.2-10.el8.noarch 2020-03-11T22:30:07.838 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mds-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:30:07.838 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:30:07.838 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mon-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:30:07.838 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-osd-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:30:07.838 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-modules-core-2:15.1.0-1941.g4acc112.el8.noarch 2020-03-11T22:30:07.838 INFO:teuthology.orchestra.run.smithi059.stdout: python3-lxml-4.2.3-1.el8.x86_64 2020-03-11T22:30:07.838 INFO:teuthology.orchestra.run.smithi059.stdout: python3-mako-1.0.6-13.el8.noarch 2020-03-11T22:30:07.838 INFO:teuthology.orchestra.run.smithi059.stdout: python3-markupsafe-0.23-19.el8.x86_64 2020-03-11T22:30:07.839 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pytz-2017.2-9.el8.noarch 2020-03-11T22:30:07.839 INFO:teuthology.orchestra.run.smithi059.stdout: python3-webencodings-0.5.1-6.el8.noarch 2020-03-11T22:30:07.839 INFO:teuthology.orchestra.run.smithi059.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2020-03-11T22:30:07.839 INFO:teuthology.orchestra.run.smithi059.stdout: python3-chardet-3.0.4-7.el8.noarch 2020-03-11T22:30:07.839 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pysocks-1.6.8-3.el8.noarch 2020-03-11T22:30:07.839 INFO:teuthology.orchestra.run.smithi059.stdout: python3-requests-2.20.0-2.1.el8_1.noarch 2020-03-11T22:30:07.839 INFO:teuthology.orchestra.run.smithi059.stdout: python3-urllib3-1.24.2-2.el8.noarch 2020-03-11T22:30:07.840 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cheroot-8.2.1-1.el8.noarch 2020-03-11T22:30:07.840 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2020-03-11T22:30:07.840 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-6.2-6.el8.noarch 2020-03-11T22:30:07.840 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2020-03-11T22:30:07.840 INFO:teuthology.orchestra.run.smithi059.stdout: python3-logutils-0.3.5-11.el8.noarch 2020-03-11T22:30:07.840 INFO:teuthology.orchestra.run.smithi059.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2020-03-11T22:30:07.840 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pecan-1.3.2-9.el8.noarch 2020-03-11T22:30:07.840 INFO:teuthology.orchestra.run.smithi059.stdout: python3-portend-2.6-1.el8.noarch 2020-03-11T22:30:07.841 INFO:teuthology.orchestra.run.smithi059.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2020-03-11T22:30:07.841 INFO:teuthology.orchestra.run.smithi059.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2020-03-11T22:30:07.841 INFO:teuthology.orchestra.run.smithi059.stdout: python3-tempora-1.14.1-5.el8.noarch 2020-03-11T22:30:07.841 INFO:teuthology.orchestra.run.smithi059.stdout: python3-trustme-0.5.2-4.el8.noarch 2020-03-11T22:30:07.842 INFO:teuthology.orchestra.run.smithi059.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2020-03-11T22:30:07.842 INFO:teuthology.orchestra.run.smithi059.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2020-03-11T22:30:07.842 INFO:teuthology.orchestra.run.smithi059.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2020-03-11T22:30:07.842 INFO:teuthology.orchestra.run.smithi059.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2020-03-11T22:30:07.842 INFO:teuthology.orchestra.run.smithi059.stdout: python3-webob-1.8.5-1.el8.1.noarch 2020-03-11T22:30:07.843 INFO:teuthology.orchestra.run.smithi059.stdout: python3-webtest-2.0.33-1.el8.noarch 2020-03-11T22:30:07.843 INFO:teuthology.orchestra.run.smithi059.stdout: 2020-03-11T22:30:07.843 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2020-03-11T22:30:08.138 INFO:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-base 2020-03-11T22:30:08.568 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:19 ago on Wed 11 Mar 2020 10:28:49 PM UTC. 2020-03-11T22:30:08.837 INFO:teuthology.orchestra.run.smithi059.stdout:Package ceph-base-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-11T22:30:08.948 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2020-03-11T22:30:08.948 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2020-03-11T22:30:08.948 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2020-03-11T22:30:09.067 INFO:teuthology.orchestra.run.smithi059:> sudo yum -y install cephadm 2020-03-11T22:30:09.495 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:20 ago on Wed 11 Mar 2020 10:28:49 PM UTC. 2020-03-11T22:30:09.924 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2020-03-11T22:30:09.924 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2020-03-11T22:30:09.924 INFO:teuthology.orchestra.run.smithi059.stdout: Package Architecture Version Repository Size 2020-03-11T22:30:09.925 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2020-03-11T22:30:09.925 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2020-03-11T22:30:09.925 INFO:teuthology.orchestra.run.smithi059.stdout: cephadm x86_64 2:15.1.0-1941.g4acc112.el8 ceph 36 k 2020-03-11T22:30:09.925 INFO:teuthology.orchestra.run.smithi059.stdout: 2020-03-11T22:30:09.925 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2020-03-11T22:30:09.925 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2020-03-11T22:30:09.925 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2020-03-11T22:30:09.926 INFO:teuthology.orchestra.run.smithi059.stdout: 2020-03-11T22:30:09.926 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 36 k 2020-03-11T22:30:09.926 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 129 k 2020-03-11T22:30:09.926 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2020-03-11T22:30:10.117 INFO:teuthology.orchestra.run.smithi059.stdout:cephadm-15.1.0-1941.g4acc112.el8.x86_64.rpm 186 kB/s | 36 kB 00:00 2020-03-11T22:30:10.118 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:30:10.118 INFO:teuthology.orchestra.run.smithi059.stdout:Total 185 kB/s | 36 kB 00:00 2020-03-11T22:30:10.120 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2020-03-11T22:30:10.128 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2020-03-11T22:30:10.128 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2020-03-11T22:30:10.130 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2020-03-11T22:30:10.131 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2020-03-11T22:30:10.178 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2020-03-11T22:30:11.615 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: cephadm-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:30:11.891 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : cephadm-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:30:12.508 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : cephadm-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:30:12.508 INFO:teuthology.orchestra.run.smithi059.stdout: 2020-03-11T22:30:12.508 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2020-03-11T22:30:12.509 INFO:teuthology.orchestra.run.smithi059.stdout: cephadm-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:30:12.509 INFO:teuthology.orchestra.run.smithi059.stdout: 2020-03-11T22:30:12.509 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2020-03-11T22:30:14.046 INFO:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-immutable-object-cache 2020-03-11T22:30:14.485 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:25 ago on Wed 11 Mar 2020 10:28:49 PM UTC. 2020-03-11T22:30:14.873 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2020-03-11T22:30:14.874 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2020-03-11T22:30:14.874 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repo Size 2020-03-11T22:30:14.874 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2020-03-11T22:30:14.874 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2020-03-11T22:30:14.874 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-immutable-object-cache x86_64 2:15.1.0-1941.g4acc112.el8 ceph 203 k 2020-03-11T22:30:14.874 INFO:teuthology.orchestra.run.smithi059.stdout: 2020-03-11T22:30:14.875 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2020-03-11T22:30:14.875 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2020-03-11T22:30:14.875 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2020-03-11T22:30:14.875 INFO:teuthology.orchestra.run.smithi059.stdout: 2020-03-11T22:30:14.875 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 203 k 2020-03-11T22:30:14.875 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 743 k 2020-03-11T22:30:14.876 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2020-03-11T22:30:15.136 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-immutable-object-cache-15.1.0-1941.g4acc11 777 kB/s | 203 kB 00:00 2020-03-11T22:30:15.137 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:30:15.137 INFO:teuthology.orchestra.run.smithi059.stdout:Total 773 kB/s | 203 kB 00:00 2020-03-11T22:30:15.139 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2020-03-11T22:30:15.153 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2020-03-11T22:30:15.153 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2020-03-11T22:30:15.203 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2020-03-11T22:30:15.203 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2020-03-11T22:30:15.465 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2020-03-11T22:30:15.607 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-immutable-object-cache-2:15.1.0-1941.g4acc112.e 1/1 2020-03-11T22:30:15.950 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-immutable-object-cache-2:15.1.0-1941.g4acc112.e 1/1 2020-03-11T22:30:16.207 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-immutable-object-cache-2:15.1.0-1941.g4acc112.e 1/1 2020-03-11T22:30:16.208 INFO:teuthology.orchestra.run.smithi059.stdout: 2020-03-11T22:30:16.208 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2020-03-11T22:30:16.208 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-immutable-object-cache-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:30:16.208 INFO:teuthology.orchestra.run.smithi059.stdout: 2020-03-11T22:30:16.208 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2020-03-11T22:30:16.465 INFO:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-mgr 2020-03-11T22:30:16.898 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:27 ago on Wed 11 Mar 2020 10:28:49 PM UTC. 2020-03-11T22:30:17.211 INFO:teuthology.orchestra.run.smithi059.stdout:Package ceph-mgr-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-11T22:30:17.323 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2020-03-11T22:30:17.324 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2020-03-11T22:30:17.324 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2020-03-11T22:30:17.459 INFO:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-mgr-dashboard 2020-03-11T22:30:17.881 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:28 ago on Wed 11 Mar 2020 10:28:49 PM UTC. 2020-03-11T22:30:18.281 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2020-03-11T22:30:18.282 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2020-03-11T22:30:18.282 INFO:teuthology.orchestra.run.smithi059.stdout: Package ArchVersion Repository Size 2020-03-11T22:30:18.282 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2020-03-11T22:30:18.283 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2020-03-11T22:30:18.283 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-dashboard noarch2:15.1.0-1941.g4acc112.el8 ceph-noarch 3.9 M 2020-03-11T22:30:18.283 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2020-03-11T22:30:18.283 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-grafana-dashboards noarch2:15.1.0-1941.g4acc112.el8 ceph-noarch 22 k 2020-03-11T22:30:18.283 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jwt noarch1.6.1-2.el8 CentOS-Base 43 k 2020-03-11T22:30:18.283 INFO:teuthology.orchestra.run.smithi059.stdout: python3-repoze-lru noarch0.7-6.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 34 k 2020-03-11T22:30:18.284 INFO:teuthology.orchestra.run.smithi059.stdout: python3-routes noarch2.4.1-12.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 197 k 2020-03-11T22:30:18.284 INFO:teuthology.orchestra.run.smithi059.stdout: 2020-03-11T22:30:18.284 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2020-03-11T22:30:18.284 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2020-03-11T22:30:18.284 INFO:teuthology.orchestra.run.smithi059.stdout:Install 5 Packages 2020-03-11T22:30:18.284 INFO:teuthology.orchestra.run.smithi059.stdout: 2020-03-11T22:30:18.285 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 4.2 M 2020-03-11T22:30:18.285 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 19 M 2020-03-11T22:30:18.285 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2020-03-11T22:30:18.316 INFO:teuthology.orchestra.run.smithi059.stdout:(1/5): python3-jwt-1.6.1-2.el8.noarch.rpm 1.3 MB/s | 43 kB 00:00 2020-03-11T22:30:18.365 INFO:teuthology.orchestra.run.smithi059.stdout:(2/5): python3-repoze-lru-0.7-6.el8.noarch.rpm 690 kB/s | 34 kB 00:00 2020-03-11T22:30:18.442 INFO:teuthology.orchestra.run.smithi059.stdout:(3/5): python3-routes-2.4.1-12.el8.noarch.rpm 2.5 MB/s | 197 kB 00:00 2020-03-11T22:30:18.473 INFO:teuthology.orchestra.run.smithi059.stdout:(4/5): ceph-grafana-dashboards-15.1.0-1941.g4ac 119 kB/s | 22 kB 00:00 2020-03-11T22:30:18.723 INFO:teuthology.orchestra.run.smithi059.stdout:(5/5): ceph-mgr-dashboard-15.1.0-1941.g4acc112. 8.8 MB/s | 3.9 MB 00:00 2020-03-11T22:30:18.726 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:30:18.726 INFO:teuthology.orchestra.run.smithi059.stdout:Total 9.4 MB/s | 4.2 MB 00:00 2020-03-11T22:30:18.731 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2020-03-11T22:30:18.740 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2020-03-11T22:30:18.740 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2020-03-11T22:30:18.891 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2020-03-11T22:30:18.891 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2020-03-11T22:30:19.129 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2020-03-11T22:30:19.340 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/5 2020-03-11T22:30:19.584 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/5 2020-03-11T22:30:19.789 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/5 2020-03-11T22:30:20.829 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-grafana-dashboards-2:15.1.0-1941.g4acc112.el8.n 4/5 2020-03-11T22:30:21.098 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-dashboard-2:15.1.0-1941.g4acc112.el8.noarch 5/5 2020-03-11T22:30:21.308 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-dashboard-2:15.1.0-1941.g4acc112.el8.noarch 5/5 2020-03-11T22:30:21.309 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-grafana-dashboards-2:15.1.0-1941.g4acc112.el8.n 1/5 2020-03-11T22:30:21.309 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-dashboard-2:15.1.0-1941.g4acc112.el8.noarch 2/5 2020-03-11T22:30:21.309 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 3/5 2020-03-11T22:30:21.309 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/5 2020-03-11T22:30:21.569 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/5 2020-03-11T22:30:21.570 INFO:teuthology.orchestra.run.smithi059.stdout: 2020-03-11T22:30:21.570 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2020-03-11T22:30:21.570 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-dashboard-2:15.1.0-1941.g4acc112.el8.noarch 2020-03-11T22:30:21.570 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-grafana-dashboards-2:15.1.0-1941.g4acc112.el8.noarch 2020-03-11T22:30:21.570 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jwt-1.6.1-2.el8.noarch 2020-03-11T22:30:21.570 INFO:teuthology.orchestra.run.smithi059.stdout: python3-repoze-lru-0.7-6.el8.noarch 2020-03-11T22:30:21.571 INFO:teuthology.orchestra.run.smithi059.stdout: python3-routes-2.4.1-12.el8.noarch 2020-03-11T22:30:21.571 INFO:teuthology.orchestra.run.smithi059.stdout: 2020-03-11T22:30:21.571 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2020-03-11T22:30:21.724 INFO:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-mgr-diskprediction-local 2020-03-11T22:30:22.150 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:33 ago on Wed 11 Mar 2020 10:28:49 PM UTC. 2020-03-11T22:30:22.558 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2020-03-11T22:30:22.559 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2020-03-11T22:30:22.560 INFO:teuthology.orchestra.run.smithi059.stdout: Package ArchVersion Repository Size 2020-03-11T22:30:22.560 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2020-03-11T22:30:22.560 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2020-03-11T22:30:22.560 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-diskprediction-local noarch2:15.1.0-1941.g4acc112.el8 ceph-noarch 8.6 M 2020-03-11T22:30:22.561 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2020-03-11T22:30:22.561 INFO:teuthology.orchestra.run.smithi059.stdout: openblas x86_640.3.3-2.el8 CentOS-AppStream 4.3 M 2020-03-11T22:30:22.561 INFO:teuthology.orchestra.run.smithi059.stdout: openblas-threads x86_640.3.3-2.el8 CentOS-AppStream 4.4 M 2020-03-11T22:30:22.561 INFO:teuthology.orchestra.run.smithi059.stdout: python3-numpy x86_641:1.14.3-9.el8 CentOS-AppStream 3.7 M 2020-03-11T22:30:22.561 INFO:teuthology.orchestra.run.smithi059.stdout: python3-numpy-f2py x86_641:1.14.3-9.el8 CentOS-AppStream 225 k 2020-03-11T22:30:22.561 INFO:teuthology.orchestra.run.smithi059.stdout: python3-scipy x86_641.0.0-20.module_el8.1.0+245+c39af44f CentOS-AppStream 15 M 2020-03-11T22:30:22.562 INFO:teuthology.orchestra.run.smithi059.stdout: libgfortran x86_648.3.1-4.5.el8 CentOS-Base 638 k 2020-03-11T22:30:22.562 INFO:teuthology.orchestra.run.smithi059.stdout: libquadmath x86_648.3.1-4.5.el8 CentOS-Base 169 k 2020-03-11T22:30:22.562 INFO:teuthology.orchestra.run.smithi059.stdout: 2020-03-11T22:30:22.562 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2020-03-11T22:30:22.562 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2020-03-11T22:30:22.562 INFO:teuthology.orchestra.run.smithi059.stdout:Install 8 Packages 2020-03-11T22:30:22.562 INFO:teuthology.orchestra.run.smithi059.stdout: 2020-03-11T22:30:22.563 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 37 M 2020-03-11T22:30:22.563 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 205 M 2020-03-11T22:30:22.563 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2020-03-11T22:30:22.673 INFO:teuthology.orchestra.run.smithi059.stdout:(1/8): openblas-0.3.3-2.el8.x86_64.rpm 39 MB/s | 4.3 MB 00:00 2020-03-11T22:30:22.732 INFO:teuthology.orchestra.run.smithi059.stdout:(2/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 64 MB/s | 3.7 MB 00:00 2020-03-11T22:30:22.737 INFO:teuthology.orchestra.run.smithi059.stdout:(3/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 42 MB/s | 225 kB 00:00 2020-03-11T22:30:22.771 INFO:teuthology.orchestra.run.smithi059.stdout:(4/8): openblas-threads-0.3.3-2.el8.x86_64.rpm 21 MB/s | 4.4 MB 00:00 2020-03-11T22:30:22.794 INFO:teuthology.orchestra.run.smithi059.stdout:(5/8): libgfortran-8.3.1-4.5.el8.x86_64.rpm 27 MB/s | 638 kB 00:00 2020-03-11T22:30:22.804 INFO:teuthology.orchestra.run.smithi059.stdout:(6/8): libquadmath-8.3.1-4.5.el8.x86_64.rpm 16 MB/s | 169 kB 00:00 2020-03-11T22:30:23.001 INFO:teuthology.orchestra.run.smithi059.stdout:(7/8): python3-scipy-1.0.0-20.module_el8.1.0+24 55 MB/s | 15 MB 00:00 2020-03-11T22:30:23.308 INFO:teuthology.orchestra.run.smithi059.stdout:(8/8): ceph-mgr-diskprediction-local-15.1.0-194 12 MB/s | 8.6 MB 00:00 2020-03-11T22:30:23.310 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:30:23.311 INFO:teuthology.orchestra.run.smithi059.stdout:Total 49 MB/s | 37 MB 00:00 2020-03-11T22:30:23.322 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2020-03-11T22:30:23.343 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2020-03-11T22:30:23.343 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2020-03-11T22:30:23.649 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2020-03-11T22:30:23.650 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2020-03-11T22:30:24.183 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2020-03-11T22:30:24.331 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libquadmath-8.3.1-4.5.el8.x86_64 1/8 2020-03-11T22:30:24.470 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: libquadmath-8.3.1-4.5.el8.x86_64 1/8 2020-03-11T22:30:24.631 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libgfortran-8.3.1-4.5.el8.x86_64 2/8 2020-03-11T22:30:25.200 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: libgfortran-8.3.1-4.5.el8.x86_64 2/8 2020-03-11T22:30:25.389 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : openblas-0.3.3-2.el8.x86_64 3/8 2020-03-11T22:30:25.964 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: openblas-0.3.3-2.el8.x86_64 3/8 2020-03-11T22:30:26.164 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : openblas-threads-0.3.3-2.el8.x86_64 4/8 2020-03-11T22:30:26.892 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: openblas-threads-0.3.3-2.el8.x86_64 4/8 2020-03-11T22:30:27.189 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2020-03-11T22:30:27.406 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2020-03-11T22:30:29.513 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2020-03-11T22:30:30.954 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-scipy-1.0.0-20.module_el8.1.0+245+c39af44f.x 7/8 2020-03-11T22:30:31.122 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-diskprediction-local-2:15.1.0-1941.g4acc112 8/8 2020-03-11T22:30:31.408 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:15.1.0-1941.g4acc112 8/8 2020-03-11T22:30:31.408 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-diskprediction-local-2:15.1.0-1941.g4acc112 1/8 2020-03-11T22:30:31.408 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : openblas-0.3.3-2.el8.x86_64 2/8 2020-03-11T22:30:31.408 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : openblas-threads-0.3.3-2.el8.x86_64 3/8 2020-03-11T22:30:31.408 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 4/8 2020-03-11T22:30:31.408 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 5/8 2020-03-11T22:30:31.409 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-scipy-1.0.0-20.module_el8.1.0+245+c39af44f.x 6/8 2020-03-11T22:30:31.409 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libgfortran-8.3.1-4.5.el8.x86_64 7/8 2020-03-11T22:30:31.727 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libquadmath-8.3.1-4.5.el8.x86_64 8/8 2020-03-11T22:30:31.727 INFO:teuthology.orchestra.run.smithi059.stdout: 2020-03-11T22:30:31.727 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2020-03-11T22:30:31.727 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-diskprediction-local-2:15.1.0-1941.g4acc112.el8.noarch 2020-03-11T22:30:31.728 INFO:teuthology.orchestra.run.smithi059.stdout: openblas-0.3.3-2.el8.x86_64 2020-03-11T22:30:31.728 INFO:teuthology.orchestra.run.smithi059.stdout: openblas-threads-0.3.3-2.el8.x86_64 2020-03-11T22:30:31.728 INFO:teuthology.orchestra.run.smithi059.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2020-03-11T22:30:31.728 INFO:teuthology.orchestra.run.smithi059.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2020-03-11T22:30:31.728 INFO:teuthology.orchestra.run.smithi059.stdout: python3-scipy-1.0.0-20.module_el8.1.0+245+c39af44f.x86_64 2020-03-11T22:30:31.728 INFO:teuthology.orchestra.run.smithi059.stdout: libgfortran-8.3.1-4.5.el8.x86_64 2020-03-11T22:30:31.728 INFO:teuthology.orchestra.run.smithi059.stdout: libquadmath-8.3.1-4.5.el8.x86_64 2020-03-11T22:30:31.729 INFO:teuthology.orchestra.run.smithi059.stdout: 2020-03-11T22:30:31.729 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2020-03-11T22:30:31.873 INFO:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-mgr-rook 2020-03-11T22:30:32.299 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:43 ago on Wed 11 Mar 2020 10:28:49 PM UTC. 2020-03-11T22:30:32.673 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2020-03-11T22:30:32.675 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2020-03-11T22:30:32.675 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2020-03-11T22:30:32.675 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2020-03-11T22:30:32.677 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2020-03-11T22:30:32.677 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-rook noarch 2:15.1.0-1941.g4acc112.el8 2020-03-11T22:30:32.677 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-noarch 24 k 2020-03-11T22:30:32.677 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2020-03-11T22:30:32.677 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jsonpatch noarch 1.21-2.el8 CentOS-AppStream 27 k 2020-03-11T22:30:32.678 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jsonpointer noarch 1.10-11.el8 CentOS-AppStream 20 k 2020-03-11T22:30:32.678 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyasn1 noarch 0.3.7-6.el8 CentOS-AppStream 126 k 2020-03-11T22:30:32.678 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 CentOS-AppStream 110 k 2020-03-11T22:30:32.678 INFO:teuthology.orchestra.run.smithi059.stdout: python3-oauthlib noarch 2.1.0-1.el8 CentOS-Base 155 k 2020-03-11T22:30:32.678 INFO:teuthology.orchestra.run.smithi059.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 CentOS-Base 43 k 2020-03-11T22:30:32.678 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2020-03-11T22:30:32.678 INFO:teuthology.orchestra.run.smithi059.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2020-03-11T22:30:32.679 INFO:teuthology.orchestra.run.smithi059.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2020-03-11T22:30:32.679 INFO:teuthology.orchestra.run.smithi059.stdout: python3-kubernetes noarch 10.0.1-2.el8 epel 1.1 M 2020-03-11T22:30:32.679 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rsa noarch 3.4.2-12.el8 epel 73 k 2020-03-11T22:30:32.679 INFO:teuthology.orchestra.run.smithi059.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2020-03-11T22:30:32.679 INFO:teuthology.orchestra.run.smithi059.stdout: 2020-03-11T22:30:32.679 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2020-03-11T22:30:32.679 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2020-03-11T22:30:32.679 INFO:teuthology.orchestra.run.smithi059.stdout:Install 13 Packages 2020-03-11T22:30:32.680 INFO:teuthology.orchestra.run.smithi059.stdout: 2020-03-11T22:30:32.680 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 1.9 M 2020-03-11T22:30:32.680 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 23 M 2020-03-11T22:30:32.681 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2020-03-11T22:30:32.709 INFO:teuthology.orchestra.run.smithi059.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 879 kB/s | 27 kB 00:00 2020-03-11T22:30:32.713 INFO:teuthology.orchestra.run.smithi059.stdout:(2/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 33 MB/s | 126 kB 00:00 2020-03-11T22:30:32.713 INFO:teuthology.orchestra.run.smithi059.stdout:(3/13): python3-jsonpointer-1.10-11.el8.noarch. 578 kB/s | 20 kB 00:00 2020-03-11T22:30:32.720 INFO:teuthology.orchestra.run.smithi059.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 15 MB/s | 110 kB 00:00 2020-03-11T22:30:32.722 INFO:teuthology.orchestra.run.smithi059.stdout:(5/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 17 MB/s | 155 kB 00:00 2020-03-11T22:30:32.723 INFO:teuthology.orchestra.run.smithi059.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 21 MB/s | 43 kB 00:00 2020-03-11T22:30:32.770 INFO:teuthology.orchestra.run.smithi059.stdout:(7/13): python3-certifi-2018.10.15-7.el8.noarch 347 kB/s | 16 kB 00:00 2020-03-11T22:30:32.779 INFO:teuthology.orchestra.run.smithi059.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 579 kB/s | 33 kB 00:00 2020-03-11T22:30:32.816 INFO:teuthology.orchestra.run.smithi059.stdout:(9/13): python3-google-auth-1.1.1-10.el8.noarch 1.7 MB/s | 81 kB 00:00 2020-03-11T22:30:32.832 INFO:teuthology.orchestra.run.smithi059.stdout:(10/13): python3-rsa-3.4.2-12.el8.noarch.rpm 4.5 MB/s | 73 kB 00:00 2020-03-11T22:30:32.848 INFO:teuthology.orchestra.run.smithi059.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 3.8 MB/s | 61 kB 00:00 2020-03-11T22:30:32.867 INFO:teuthology.orchestra.run.smithi059.stdout:(12/13): ceph-mgr-rook-15.1.0-1941.g4acc112.el8 129 kB/s | 24 kB 00:00 2020-03-11T22:30:32.895 INFO:teuthology.orchestra.run.smithi059.stdout:(13/13): python3-kubernetes-10.0.1-2.el8.noarch 9.6 MB/s | 1.1 MB 00:00 2020-03-11T22:30:32.896 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:30:32.896 INFO:teuthology.orchestra.run.smithi059.stdout:Total 8.5 MB/s | 1.9 MB 00:00 2020-03-11T22:30:32.904 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2020-03-11T22:30:32.919 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2020-03-11T22:30:32.920 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2020-03-11T22:30:32.993 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2020-03-11T22:30:32.993 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2020-03-11T22:30:33.479 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2020-03-11T22:30:33.745 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2020-03-11T22:30:33.997 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2020-03-11T22:30:34.201 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-rsa-3.4.2-12.el8.noarch 3/13 2020-03-11T22:30:34.429 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2020-03-11T22:30:34.803 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2020-03-11T22:30:35.900 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2020-03-11T22:30:36.197 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2020-03-11T22:30:36.429 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2020-03-11T22:30:37.558 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2020-03-11T22:30:37.804 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-kubernetes-10.0.1-2.el8.noarch 10/13 2020-03-11T22:30:38.046 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2020-03-11T22:30:38.205 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2020-03-11T22:30:38.387 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-rook-2:15.1.0-1941.g4acc112.el8.noarch 13/13 2020-03-11T22:30:38.614 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-rook-2:15.1.0-1941.g4acc112.el8.noarch 13/13 2020-03-11T22:30:38.614 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-rook-2:15.1.0-1941.g4acc112.el8.noarch 1/13 2020-03-11T22:30:38.614 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2020-03-11T22:30:38.615 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2020-03-11T22:30:38.615 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2020-03-11T22:30:38.615 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2020-03-11T22:30:38.615 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2020-03-11T22:30:38.615 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2020-03-11T22:30:38.615 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2020-03-11T22:30:38.615 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2020-03-11T22:30:38.615 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2020-03-11T22:30:38.616 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-kubernetes-10.0.1-2.el8.noarch 11/13 2020-03-11T22:30:38.616 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-rsa-3.4.2-12.el8.noarch 12/13 2020-03-11T22:30:38.955 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2020-03-11T22:30:38.955 INFO:teuthology.orchestra.run.smithi059.stdout: 2020-03-11T22:30:38.955 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2020-03-11T22:30:38.955 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-rook-2:15.1.0-1941.g4acc112.el8.noarch 2020-03-11T22:30:38.955 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jsonpatch-1.21-2.el8.noarch 2020-03-11T22:30:38.955 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jsonpointer-1.10-11.el8.noarch 2020-03-11T22:30:38.956 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2020-03-11T22:30:38.956 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2020-03-11T22:30:38.956 INFO:teuthology.orchestra.run.smithi059.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2020-03-11T22:30:38.956 INFO:teuthology.orchestra.run.smithi059.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2020-03-11T22:30:38.956 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cachetools-3.1.1-4.el8.noarch 2020-03-11T22:30:38.956 INFO:teuthology.orchestra.run.smithi059.stdout: python3-certifi-2018.10.15-7.el8.noarch 2020-03-11T22:30:38.956 INFO:teuthology.orchestra.run.smithi059.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2020-03-11T22:30:38.956 INFO:teuthology.orchestra.run.smithi059.stdout: python3-kubernetes-10.0.1-2.el8.noarch 2020-03-11T22:30:38.957 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rsa-3.4.2-12.el8.noarch 2020-03-11T22:30:38.957 INFO:teuthology.orchestra.run.smithi059.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2020-03-11T22:30:38.957 INFO:teuthology.orchestra.run.smithi059.stdout: 2020-03-11T22:30:38.957 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2020-03-11T22:30:39.080 INFO:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-mgr-cephadm 2020-03-11T22:30:39.512 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:50 ago on Wed 11 Mar 2020 10:28:49 PM UTC. 2020-03-11T22:30:39.894 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2020-03-11T22:30:39.896 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2020-03-11T22:30:39.896 INFO:teuthology.orchestra.run.smithi059.stdout: Package ArchVersion Repository Size 2020-03-11T22:30:39.896 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2020-03-11T22:30:39.896 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2020-03-11T22:30:39.896 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-cephadm noarch2:15.1.0-1941.g4acc112.el8 ceph-noarch 36 k 2020-03-11T22:30:39.896 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2020-03-11T22:30:39.897 INFO:teuthology.orchestra.run.smithi059.stdout: python3-execnet noarch1.7.1-1.el8.2 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 356 k 2020-03-11T22:30:39.897 INFO:teuthology.orchestra.run.smithi059.stdout: python3-remoto noarch1.1.4-4.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 48 k 2020-03-11T22:30:39.897 INFO:teuthology.orchestra.run.smithi059.stdout: python3-apipkg noarch1.5-6.el8 epel 21 k 2020-03-11T22:30:39.897 INFO:teuthology.orchestra.run.smithi059.stdout: 2020-03-11T22:30:39.897 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2020-03-11T22:30:39.897 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2020-03-11T22:30:39.897 INFO:teuthology.orchestra.run.smithi059.stdout:Install 4 Packages 2020-03-11T22:30:39.898 INFO:teuthology.orchestra.run.smithi059.stdout: 2020-03-11T22:30:39.898 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 461 k 2020-03-11T22:30:39.898 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 1.4 M 2020-03-11T22:30:39.898 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2020-03-11T22:30:39.951 INFO:teuthology.orchestra.run.smithi059.stdout:(1/4): python3-remoto-1.1.4-4.el8.noarch.rpm 890 kB/s | 48 kB 00:00 2020-03-11T22:30:39.963 INFO:teuthology.orchestra.run.smithi059.stdout:(2/4): python3-execnet-1.7.1-1.el8.2.noarch.rpm 5.3 MB/s | 356 kB 00:00 2020-03-11T22:30:40.009 INFO:teuthology.orchestra.run.smithi059.stdout:(3/4): python3-apipkg-1.5-6.el8.noarch.rpm 366 kB/s | 21 kB 00:00 2020-03-11T22:30:40.101 INFO:teuthology.orchestra.run.smithi059.stdout:(4/4): ceph-mgr-cephadm-15.1.0-1941.g4acc112.el 178 kB/s | 36 kB 00:00 2020-03-11T22:30:40.103 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:30:40.103 INFO:teuthology.orchestra.run.smithi059.stdout:Total 2.2 MB/s | 461 kB 00:00 2020-03-11T22:30:40.107 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2020-03-11T22:30:40.114 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2020-03-11T22:30:40.114 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2020-03-11T22:30:40.146 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2020-03-11T22:30:40.147 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2020-03-11T22:30:40.320 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2020-03-11T22:30:40.568 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/4 2020-03-11T22:30:40.834 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-execnet-1.7.1-1.el8.2.noarch 2/4 2020-03-11T22:30:41.018 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-remoto-1.1.4-4.el8.noarch 3/4 2020-03-11T22:30:41.212 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-cephadm-2:15.1.0-1941.g4acc112.el8.noarch 4/4 2020-03-11T22:30:41.763 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-cephadm-2:15.1.0-1941.g4acc112.el8.noarch 4/4 2020-03-11T22:30:41.763 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-cephadm-2:15.1.0-1941.g4acc112.el8.noarch 1/4 2020-03-11T22:30:41.763 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-execnet-1.7.1-1.el8.2.noarch 2/4 2020-03-11T22:30:41.763 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-remoto-1.1.4-4.el8.noarch 3/4 2020-03-11T22:30:42.013 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/4 2020-03-11T22:30:42.013 INFO:teuthology.orchestra.run.smithi059.stdout: 2020-03-11T22:30:42.013 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2020-03-11T22:30:42.013 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-cephadm-2:15.1.0-1941.g4acc112.el8.noarch 2020-03-11T22:30:42.013 INFO:teuthology.orchestra.run.smithi059.stdout: python3-execnet-1.7.1-1.el8.2.noarch 2020-03-11T22:30:42.014 INFO:teuthology.orchestra.run.smithi059.stdout: python3-remoto-1.1.4-4.el8.noarch 2020-03-11T22:30:42.014 INFO:teuthology.orchestra.run.smithi059.stdout: python3-apipkg-1.5-6.el8.noarch 2020-03-11T22:30:42.014 INFO:teuthology.orchestra.run.smithi059.stdout: 2020-03-11T22:30:42.014 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2020-03-11T22:30:42.145 INFO:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-fuse 2020-03-11T22:30:42.572 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:53 ago on Wed 11 Mar 2020 10:28:49 PM UTC. 2020-03-11T22:30:42.960 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2020-03-11T22:30:42.960 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2020-03-11T22:30:42.961 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2020-03-11T22:30:42.961 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2020-03-11T22:30:42.961 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2020-03-11T22:30:42.961 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-fuse x86_64 2:15.1.0-1941.g4acc112.el8 ceph 572 k 2020-03-11T22:30:42.961 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2020-03-11T22:30:42.961 INFO:teuthology.orchestra.run.smithi059.stdout: fuse x86_64 2.9.7-12.el8 CentOS-Base 82 k 2020-03-11T22:30:42.962 INFO:teuthology.orchestra.run.smithi059.stdout: fuse-common x86_64 3.2.1-12.el8 CentOS-Base 21 k 2020-03-11T22:30:42.962 INFO:teuthology.orchestra.run.smithi059.stdout: 2020-03-11T22:30:42.962 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2020-03-11T22:30:42.962 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2020-03-11T22:30:42.962 INFO:teuthology.orchestra.run.smithi059.stdout:Install 3 Packages 2020-03-11T22:30:42.962 INFO:teuthology.orchestra.run.smithi059.stdout: 2020-03-11T22:30:42.963 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 676 k 2020-03-11T22:30:42.963 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 2.1 M 2020-03-11T22:30:42.963 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2020-03-11T22:30:42.994 INFO:teuthology.orchestra.run.smithi059.stdout:(1/3): fuse-2.9.7-12.el8.x86_64.rpm 2.5 MB/s | 82 kB 00:00 2020-03-11T22:30:43.006 INFO:teuthology.orchestra.run.smithi059.stdout:(2/3): fuse-common-3.2.1-12.el8.x86_64.rpm 485 kB/s | 21 kB 00:00 2020-03-11T22:30:43.513 INFO:teuthology.orchestra.run.smithi059.stdout:(3/3): ceph-fuse-15.1.0-1941.g4acc112.el8.x86_6 1.0 MB/s | 572 kB 00:00 2020-03-11T22:30:43.514 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:30:43.515 INFO:teuthology.orchestra.run.smithi059.stdout:Total 1.2 MB/s | 676 kB 00:00 2020-03-11T22:30:43.519 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2020-03-11T22:30:43.543 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2020-03-11T22:30:43.543 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2020-03-11T22:30:43.615 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2020-03-11T22:30:43.615 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2020-03-11T22:30:43.787 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2020-03-11T22:30:43.788 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : fuse-common-3.2.1-12.el8.x86_64 1/3 2020-03-11T22:30:43.788 INFO:teuthology.orchestra.run.smithi059.stdout:warning: /etc/fuse.conf created as /etc/fuse.conf.rpmnew 2020-03-11T22:30:43.788 INFO:teuthology.orchestra.run.smithi059.stdout: 2020-03-11T22:30:44.214 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : fuse-2.9.7-12.el8.x86_64 2/3 2020-03-11T22:30:44.394 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-fuse-2:15.1.0-1941.g4acc112.el8.x86_64 3/3 2020-03-11T22:30:44.701 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-fuse-2:15.1.0-1941.g4acc112.el8.x86_64 3/3 2020-03-11T22:30:44.702 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-fuse-2:15.1.0-1941.g4acc112.el8.x86_64 1/3 2020-03-11T22:30:44.702 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/3 2020-03-11T22:30:44.989 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : fuse-common-3.2.1-12.el8.x86_64 3/3 2020-03-11T22:30:44.989 INFO:teuthology.orchestra.run.smithi059.stdout: 2020-03-11T22:30:44.989 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2020-03-11T22:30:44.989 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-fuse-2:15.1.0-1941.g4acc112.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2020-03-11T22:30:44.989 INFO:teuthology.orchestra.run.smithi059.stdout: fuse-common-3.2.1-12.el8.x86_64 2020-03-11T22:30:44.990 INFO:teuthology.orchestra.run.smithi059.stdout: 2020-03-11T22:30:44.990 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2020-03-11T22:30:45.300 INFO:teuthology.orchestra.run.smithi059:> sudo yum -y install librados-devel 2020-03-11T22:30:45.728 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:56 ago on Wed 11 Mar 2020 10:28:49 PM UTC. 2020-03-11T22:30:46.189 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2020-03-11T22:30:46.189 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2020-03-11T22:30:46.190 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repo Size 2020-03-11T22:30:46.190 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2020-03-11T22:30:46.190 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2020-03-11T22:30:46.190 INFO:teuthology.orchestra.run.smithi059.stdout: librados-devel x86_64 2:15.1.0-1941.g4acc112.el8 ceph 141 k 2020-03-11T22:30:46.190 INFO:teuthology.orchestra.run.smithi059.stdout: 2020-03-11T22:30:46.191 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2020-03-11T22:30:46.191 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2020-03-11T22:30:46.191 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2020-03-11T22:30:46.191 INFO:teuthology.orchestra.run.smithi059.stdout: 2020-03-11T22:30:46.192 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 141 k 2020-03-11T22:30:46.192 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 487 k 2020-03-11T22:30:46.192 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2020-03-11T22:30:46.452 INFO:teuthology.orchestra.run.smithi059.stdout:librados-devel-15.1.0-1941.g4acc112.el8.x86_64. 539 kB/s | 141 kB 00:00 2020-03-11T22:30:46.452 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:30:46.453 INFO:teuthology.orchestra.run.smithi059.stdout:Total 537 kB/s | 141 kB 00:00 2020-03-11T22:30:46.454 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2020-03-11T22:30:46.462 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2020-03-11T22:30:46.462 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2020-03-11T22:30:46.516 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2020-03-11T22:30:46.516 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2020-03-11T22:30:46.625 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2020-03-11T22:30:46.868 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : librados-devel-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:30:47.233 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librados-devel-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:30:47.506 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librados-devel-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:30:47.506 INFO:teuthology.orchestra.run.smithi059.stdout: 2020-03-11T22:30:47.506 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2020-03-11T22:30:47.506 INFO:teuthology.orchestra.run.smithi059.stdout: librados-devel-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:30:47.506 INFO:teuthology.orchestra.run.smithi059.stdout: 2020-03-11T22:30:47.506 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2020-03-11T22:30:47.759 INFO:teuthology.orchestra.run.smithi059:> sudo yum -y install libcephfs2 2020-03-11T22:30:48.187 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:59 ago on Wed 11 Mar 2020 10:28:49 PM UTC. 2020-03-11T22:30:48.467 INFO:teuthology.orchestra.run.smithi059.stdout:Package libcephfs2-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-11T22:30:48.582 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2020-03-11T22:30:48.583 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2020-03-11T22:30:48.583 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2020-03-11T22:30:48.693 INFO:teuthology.orchestra.run.smithi059:> sudo yum -y install libcephfs-devel 2020-03-11T22:30:49.119 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:00 ago on Wed 11 Mar 2020 10:28:49 PM UTC. 2020-03-11T22:30:49.504 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2020-03-11T22:30:49.505 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2020-03-11T22:30:49.505 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repo Size 2020-03-11T22:30:49.505 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2020-03-11T22:30:49.505 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2020-03-11T22:30:49.505 INFO:teuthology.orchestra.run.smithi059.stdout: libcephfs-devel x86_64 2:15.1.0-1941.g4acc112.el8 ceph 21 k 2020-03-11T22:30:49.506 INFO:teuthology.orchestra.run.smithi059.stdout: 2020-03-11T22:30:49.506 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2020-03-11T22:30:49.506 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2020-03-11T22:30:49.506 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2020-03-11T22:30:49.506 INFO:teuthology.orchestra.run.smithi059.stdout: 2020-03-11T22:30:49.507 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 21 k 2020-03-11T22:30:49.507 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 76 k 2020-03-11T22:30:49.507 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2020-03-11T22:30:49.813 INFO:teuthology.orchestra.run.smithi059.stdout:libcephfs-devel-15.1.0-1941.g4acc112.el8.x86_64 69 kB/s | 21 kB 00:00 2020-03-11T22:30:49.814 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:30:49.814 INFO:teuthology.orchestra.run.smithi059.stdout:Total 69 kB/s | 21 kB 00:00 2020-03-11T22:30:49.816 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2020-03-11T22:30:49.821 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2020-03-11T22:30:49.821 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2020-03-11T22:30:49.827 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2020-03-11T22:30:49.827 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2020-03-11T22:30:49.873 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2020-03-11T22:30:50.059 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libcephfs-devel-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:30:50.243 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: libcephfs-devel-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:30:50.482 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libcephfs-devel-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:30:50.483 INFO:teuthology.orchestra.run.smithi059.stdout: 2020-03-11T22:30:50.483 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2020-03-11T22:30:50.483 INFO:teuthology.orchestra.run.smithi059.stdout: libcephfs-devel-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:30:50.483 INFO:teuthology.orchestra.run.smithi059.stdout: 2020-03-11T22:30:50.483 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2020-03-11T22:30:50.634 INFO:teuthology.orchestra.run.smithi059:> sudo yum -y install librados2 2020-03-11T22:30:51.067 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:02 ago on Wed 11 Mar 2020 10:28:49 PM UTC. 2020-03-11T22:30:51.323 INFO:teuthology.orchestra.run.smithi059.stdout:Package librados2-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-11T22:30:51.436 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2020-03-11T22:30:51.437 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2020-03-11T22:30:51.437 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2020-03-11T22:30:51.554 INFO:teuthology.orchestra.run.smithi059:> sudo yum -y install librbd1 2020-03-11T22:30:51.983 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:02 ago on Wed 11 Mar 2020 10:28:49 PM UTC. 2020-03-11T22:30:52.283 INFO:teuthology.orchestra.run.smithi059.stdout:Package librbd1-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-11T22:30:52.396 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2020-03-11T22:30:52.397 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2020-03-11T22:30:52.397 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2020-03-11T22:30:52.500 INFO:teuthology.orchestra.run.smithi059:> sudo yum -y install python3-rados 2020-03-11T22:30:52.932 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:03 ago on Wed 11 Mar 2020 10:28:49 PM UTC. 2020-03-11T22:30:53.214 INFO:teuthology.orchestra.run.smithi059.stdout:Package python3-rados-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-11T22:30:53.325 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2020-03-11T22:30:53.326 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2020-03-11T22:30:53.327 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2020-03-11T22:30:53.457 INFO:teuthology.orchestra.run.smithi059:> sudo yum -y install python3-rgw 2020-03-11T22:30:53.884 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:04 ago on Wed 11 Mar 2020 10:28:49 PM UTC. 2020-03-11T22:30:54.166 INFO:teuthology.orchestra.run.smithi059.stdout:Package python3-rgw-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-11T22:30:54.277 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2020-03-11T22:30:54.279 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2020-03-11T22:30:54.279 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2020-03-11T22:30:54.386 INFO:teuthology.orchestra.run.smithi059:> sudo yum -y install python3-cephfs 2020-03-11T22:30:54.813 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:05 ago on Wed 11 Mar 2020 10:28:49 PM UTC. 2020-03-11T22:30:55.122 INFO:teuthology.orchestra.run.smithi059.stdout:Package python3-cephfs-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-11T22:30:55.235 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2020-03-11T22:30:55.237 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2020-03-11T22:30:55.237 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2020-03-11T22:30:55.366 INFO:teuthology.orchestra.run.smithi059:> sudo yum -y install python3-rbd 2020-03-11T22:30:55.798 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:06 ago on Wed 11 Mar 2020 10:28:49 PM UTC. 2020-03-11T22:30:56.072 INFO:teuthology.orchestra.run.smithi059.stdout:Package python3-rbd-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-11T22:30:56.182 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2020-03-11T22:30:56.184 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2020-03-11T22:30:56.184 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2020-03-11T22:30:56.299 INFO:teuthology.orchestra.run.smithi059:> sudo yum -y install rbd-fuse 2020-03-11T22:30:56.727 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:07 ago on Wed 11 Mar 2020 10:28:49 PM UTC. 2020-03-11T22:30:57.158 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2020-03-11T22:30:57.159 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2020-03-11T22:30:57.159 INFO:teuthology.orchestra.run.smithi059.stdout: Package Architecture Version Repository Size 2020-03-11T22:30:57.159 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2020-03-11T22:30:57.159 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2020-03-11T22:30:57.160 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-fuse x86_64 2:15.1.0-1941.g4acc112.el8 ceph 72 k 2020-03-11T22:30:57.160 INFO:teuthology.orchestra.run.smithi059.stdout: 2020-03-11T22:30:57.160 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2020-03-11T22:30:57.160 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2020-03-11T22:30:57.160 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2020-03-11T22:30:57.160 INFO:teuthology.orchestra.run.smithi059.stdout: 2020-03-11T22:30:57.161 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 72 k 2020-03-11T22:30:57.161 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 230 k 2020-03-11T22:30:57.161 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2020-03-11T22:30:57.358 INFO:teuthology.orchestra.run.smithi059.stdout:rbd-fuse-15.1.0-1941.g4acc112.el8.x86_64.rpm 364 kB/s | 72 kB 00:00 2020-03-11T22:30:57.359 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:30:57.359 INFO:teuthology.orchestra.run.smithi059.stdout:Total 362 kB/s | 72 kB 00:00 2020-03-11T22:30:57.361 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2020-03-11T22:30:57.373 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2020-03-11T22:30:57.373 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2020-03-11T22:30:57.426 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2020-03-11T22:30:57.426 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2020-03-11T22:30:57.533 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2020-03-11T22:30:58.510 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : rbd-fuse-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:30:59.025 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: rbd-fuse-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:30:59.306 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : rbd-fuse-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:30:59.307 INFO:teuthology.orchestra.run.smithi059.stdout: 2020-03-11T22:30:59.307 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2020-03-11T22:30:59.307 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-fuse-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:30:59.307 INFO:teuthology.orchestra.run.smithi059.stdout: 2020-03-11T22:30:59.307 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2020-03-11T22:30:59.608 INFO:teuthology.orchestra.run.smithi059:> sudo yum -y install rbd-mirror 2020-03-11T22:31:00.058 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:11 ago on Wed 11 Mar 2020 10:28:49 PM UTC. 2020-03-11T22:31:00.469 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2020-03-11T22:31:00.470 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2020-03-11T22:31:00.470 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repo Size 2020-03-11T22:31:00.470 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2020-03-11T22:31:00.471 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2020-03-11T22:31:00.471 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-mirror x86_64 2:15.1.0-1941.g4acc112.el8 ceph 2.7 M 2020-03-11T22:31:00.471 INFO:teuthology.orchestra.run.smithi059.stdout: 2020-03-11T22:31:00.471 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2020-03-11T22:31:00.471 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2020-03-11T22:31:00.471 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2020-03-11T22:31:00.472 INFO:teuthology.orchestra.run.smithi059.stdout: 2020-03-11T22:31:00.472 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 2.7 M 2020-03-11T22:31:00.472 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 12 M 2020-03-11T22:31:00.472 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2020-03-11T22:31:00.906 INFO:teuthology.orchestra.run.smithi059.stdout:rbd-mirror-15.1.0-1941.g4acc112.el8.x86_64.rpm 6.2 MB/s | 2.7 MB 00:00 2020-03-11T22:31:00.906 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:31:00.907 INFO:teuthology.orchestra.run.smithi059.stdout:Total 6.2 MB/s | 2.7 MB 00:00 2020-03-11T22:31:00.909 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2020-03-11T22:31:00.928 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2020-03-11T22:31:00.928 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2020-03-11T22:31:00.985 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2020-03-11T22:31:00.985 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2020-03-11T22:31:01.361 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2020-03-11T22:31:01.568 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : rbd-mirror-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:31:01.906 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: rbd-mirror-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:31:02.186 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : rbd-mirror-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:31:02.186 INFO:teuthology.orchestra.run.smithi059.stdout: 2020-03-11T22:31:02.186 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2020-03-11T22:31:02.186 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-mirror-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:31:02.186 INFO:teuthology.orchestra.run.smithi059.stdout: 2020-03-11T22:31:02.187 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2020-03-11T22:31:02.417 INFO:teuthology.orchestra.run.smithi059:> sudo yum -y install rbd-nbd 2020-03-11T22:31:02.851 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:13 ago on Wed 11 Mar 2020 10:28:49 PM UTC. 2020-03-11T22:31:03.266 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2020-03-11T22:31:03.267 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2020-03-11T22:31:03.268 INFO:teuthology.orchestra.run.smithi059.stdout: Package Architecture Version Repository Size 2020-03-11T22:31:03.268 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2020-03-11T22:31:03.268 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2020-03-11T22:31:03.268 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-nbd x86_64 2:15.1.0-1941.g4acc112.el8 ceph 141 k 2020-03-11T22:31:03.268 INFO:teuthology.orchestra.run.smithi059.stdout: 2020-03-11T22:31:03.268 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2020-03-11T22:31:03.269 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2020-03-11T22:31:03.269 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2020-03-11T22:31:03.269 INFO:teuthology.orchestra.run.smithi059.stdout: 2020-03-11T22:31:03.269 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 141 k 2020-03-11T22:31:03.269 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 526 k 2020-03-11T22:31:03.269 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2020-03-11T22:31:03.499 INFO:teuthology.orchestra.run.smithi059.stdout:rbd-nbd-15.1.0-1941.g4acc112.el8.x86_64.rpm 610 kB/s | 141 kB 00:00 2020-03-11T22:31:03.500 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:31:03.500 INFO:teuthology.orchestra.run.smithi059.stdout:Total 607 kB/s | 141 kB 00:00 2020-03-11T22:31:03.502 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2020-03-11T22:31:03.515 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2020-03-11T22:31:03.515 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2020-03-11T22:31:03.569 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2020-03-11T22:31:03.569 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2020-03-11T22:31:03.691 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2020-03-11T22:31:04.692 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : rbd-nbd-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:31:04.909 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: rbd-nbd-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:31:05.206 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : rbd-nbd-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:31:05.206 INFO:teuthology.orchestra.run.smithi059.stdout: 2020-03-11T22:31:05.206 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2020-03-11T22:31:05.207 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-nbd-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:31:05.207 INFO:teuthology.orchestra.run.smithi059.stdout: 2020-03-11T22:31:05.207 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2020-03-11T22:31:05.478 DEBUG:teuthology.parallel:result is None 2020-03-11T22:31:05.478 INFO:teuthology.orchestra.run.smithi059:> sudo lsb_release -is 2020-03-11T22:31:05.519 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS 2020-03-11T22:31:05.520 DEBUG:teuthology.misc:System to be installed: CentOS 2020-03-11T22:31:05.520 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=4acc1121e47039a3cdf2978ae458c10d153aa141 2020-03-11T22:31:05.751 INFO:teuthology.orchestra.run.smithi059:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2020-03-11T22:31:05.771 INFO:teuthology.orchestra.run.smithi059.stdout:15.1.0-1941.g4acc112.el8 2020-03-11T22:31:05.771 INFO:teuthology.packaging:The installed version of ceph is 15.1.0-1941.g4acc112.el8 2020-03-11T22:31:05.771 INFO:teuthology.task.install:The correct ceph version 15.1.0-1941.g4acc112 is installed. 2020-03-11T22:31:05.771 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2020-03-11T22:31:05.779 INFO:teuthology.orchestra.run.smithi059:> sudo sh -c 'cat > /home/ubuntu/cephtest/valgrind.supp' 2020-03-11T22:31:05.843 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2020-03-11T22:31:05.846 INFO:teuthology.orchestra.run.smithi059:> sudo sh -c 'cat > /usr/bin/daemon-helper' 2020-03-11T22:31:05.914 INFO:teuthology.orchestra.run.smithi059:> sudo chmod a=rx -- /usr/bin/daemon-helper 2020-03-11T22:31:05.985 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2020-03-11T22:31:05.985 INFO:teuthology.orchestra.run.smithi059:> sudo sh -c 'cat > /usr/bin/adjust-ulimits' 2020-03-11T22:31:06.070 INFO:teuthology.orchestra.run.smithi059:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2020-03-11T22:31:06.141 INFO:teuthology.run_tasks:Running task workunit... 2020-03-11T22:31:06.162 INFO:tasks.workunit:Pulling workunits from ref 4acc1121e47039a3cdf2978ae458c10d153aa141 2020-03-11T22:31:06.162 INFO:tasks.workunit:Making a separate scratch dir for every client... 2020-03-11T22:31:06.163 INFO:tasks.workunit:timeout=3h 2020-03-11T22:31:06.163 INFO:tasks.workunit:cleanup=True 2020-03-11T22:31:06.163 INFO:teuthology.orchestra.run.smithi059:> stat -- /home/ubuntu/cephtest/mnt.0 2020-03-11T22:31:06.195 INFO:teuthology.orchestra.run.smithi059.stderr:stat: cannot stat '/home/ubuntu/cephtest/mnt.0': No such file or directory 2020-03-11T22:31:06.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-03-11T22:31:06.195 INFO:teuthology.orchestra.run.smithi059:> mkdir -- /home/ubuntu/cephtest/mnt.0 2020-03-11T22:31:06.249 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2020-03-11T22:31:06.249 INFO:teuthology.orchestra.run.smithi059:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2020-03-11T22:31:06.303 INFO:teuthology.orchestra.run.smithi059:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 4acc1121e47039a3cdf2978ae458c10d153aa141 2020-03-11T22:31:06.415 INFO:tasks.workunit.client.0.smithi059.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2020-03-11T22:32:41.256 INFO:tasks.workunit.client.0.smithi059.stderr:Note: checking out '4acc1121e47039a3cdf2978ae458c10d153aa141'. 2020-03-11T22:32:41.256 INFO:tasks.workunit.client.0.smithi059.stderr: 2020-03-11T22:32:41.256 INFO:tasks.workunit.client.0.smithi059.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2020-03-11T22:32:41.257 INFO:tasks.workunit.client.0.smithi059.stderr:changes and commit them, and you can discard any commits you make in this 2020-03-11T22:32:41.257 INFO:tasks.workunit.client.0.smithi059.stderr:state without impacting any branches by performing another checkout. 2020-03-11T22:32:41.257 INFO:tasks.workunit.client.0.smithi059.stderr: 2020-03-11T22:32:41.257 INFO:tasks.workunit.client.0.smithi059.stderr:If you want to create a new branch to retain commits you create, you may 2020-03-11T22:32:41.257 INFO:tasks.workunit.client.0.smithi059.stderr:do so (now or later) by using -b with the checkout command again. Example: 2020-03-11T22:32:41.257 INFO:tasks.workunit.client.0.smithi059.stderr: 2020-03-11T22:32:41.257 INFO:tasks.workunit.client.0.smithi059.stderr: git checkout -b 2020-03-11T22:32:41.258 INFO:tasks.workunit.client.0.smithi059.stderr: 2020-03-11T22:32:41.258 INFO:tasks.workunit.client.0.smithi059.stderr:HEAD is now at 4acc1121e4 Over 30 seconds monitor pg states 2020-03-11T22:32:41.261 INFO:teuthology.orchestra.run.smithi059:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2020-03-11T22:32:41.336 DEBUG:teuthology.orchestra.remote:smithi059:/home/ubuntu/cephtest/workunits.list.client.0 is 1KB 2020-03-11T22:32:41.386 INFO:tasks.workunit:Running workunits matching osd on client.0... 2020-03-11T22:32:41.386 INFO:tasks.workunit:Running workunit osd/divergent-priors.sh... 2020-03-11T22:32:41.387 INFO:teuthology.orchestra.run.smithi059:workunit test osd/divergent-priors.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4acc1121e47039a3cdf2978ae458c10d153aa141 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh 2020-03-11T22:32:41.426 INFO:tasks.workunit.client.0.smithi059.stderr:stty: 'standard input': Inappropriate ioctl for device 2020-03-11T22:32:41.432 INFO:tasks.workunit.client.0.smithi059.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2020-03-11T22:32:41.432 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2020-03-11T22:32:41.432 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2020-03-11T22:32:41.433 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: main: export PYTHONWARNINGS=ignore 2020-03-11T22:32:41.433 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: main: PYTHONWARNINGS=ignore 2020-03-11T22:32:41.433 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: main: export CEPH_CONF=/dev/null 2020-03-11T22:32:41.433 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: main: CEPH_CONF=/dev/null 2020-03-11T22:32:41.434 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: main: unset CEPH_ARGS 2020-03-11T22:32:41.434 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: local code 2020-03-11T22:32:41.434 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: run td/divergent-priors 2020-03-11T22:32:41.434 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:21: run: local dir=td/divergent-priors 2020-03-11T22:32:41.435 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:22: run: shift 2020-03-11T22:32:41.435 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:25: run: export loglen=12 2020-03-11T22:32:41.435 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:25: run: loglen=12 2020-03-11T22:32:41.435 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:26: run: export divisor=3 2020-03-11T22:32:41.436 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:26: run: divisor=3 2020-03-11T22:32:41.436 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: expr 12 / 2 2020-03-11T22:32:41.436 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: export trim=6 2020-03-11T22:32:41.436 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: trim=6 2020-03-11T22:32:41.437 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: expr 6 / 3 2020-03-11T22:32:41.437 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: export DIVERGENT_WRITE=2 2020-03-11T22:32:41.437 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: DIVERGENT_WRITE=2 2020-03-11T22:32:41.437 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: expr 6 / 3 2020-03-11T22:32:41.437 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: export DIVERGENT_REMOVE=2 2020-03-11T22:32:41.438 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: DIVERGENT_REMOVE=2 2020-03-11T22:32:41.438 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: expr 6 / 3 2020-03-11T22:32:41.445 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: export DIVERGENT_CREATE=2 2020-03-11T22:32:41.445 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: DIVERGENT_CREATE=2 2020-03-11T22:32:41.445 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:31: run: export poolname=test 2020-03-11T22:32:41.446 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:31: run: poolname=test 2020-03-11T22:32:41.446 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:32: run: export testobjects=100 2020-03-11T22:32:41.446 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:32: run: testobjects=100 2020-03-11T22:32:41.446 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:34: run: export CEPH_MON=127.0.0.1:7115 2020-03-11T22:32:41.447 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:34: run: CEPH_MON=127.0.0.1:7115 2020-03-11T22:32:41.447 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:35: run: export CEPH_ARGS 2020-03-11T22:32:41.447 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:36: run: uuidgen 2020-03-11T22:32:41.447 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:36: run: CEPH_ARGS+='--fsid=cb7f2e42-1934-4145-a3c3-29bdd93f8adb --auth-supported=none ' 2020-03-11T22:32:41.448 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:37: run: CEPH_ARGS+='--mon-host=127.0.0.1:7115 ' 2020-03-11T22:32:41.448 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:39: run: CEPH_ARGS+='--osd_force_auth_primary_missing_objects=1000000 ' 2020-03-11T22:32:41.448 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:40: run: CEPH_ARGS+='--osd_debug_pg_log_writeout=true ' 2020-03-11T22:32:41.448 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:41: run: CEPH_ARGS+='--osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:32:41.449 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: set 2020-03-11T22:32:41.449 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2020-03-11T22:32:41.449 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: local 'funcs=TEST_divergent 2020-03-11T22:32:41.449 INFO:tasks.workunit.client.0.smithi059.stderr:TEST_divergent_2 2020-03-11T22:32:41.449 INFO:tasks.workunit.client.0.smithi059.stderr:TEST_divergent_3 2020-03-11T22:32:41.450 INFO:tasks.workunit.client.0.smithi059.stderr:TEST_divergent_ec' 2020-03-11T22:32:41.450 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2020-03-11T22:32:41.450 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2020-03-11T22:32:41.450 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2020-03-11T22:32:41.450 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2020-03-11T22:32:41.451 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-11T22:32:41.451 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:32:41.451 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-11T22:32:41.451 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:32:41.451 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:32:41.454 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:32:41.454 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:32:41.454 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:32:41.455 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:32:41.455 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:32:41.455 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:32:41.455 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:32:41.455 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:32:41.455 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:32:41.456 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:32:41.456 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:32:41.456 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:32:41.456 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:32:41.456 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:32:41.456 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:32:41.460 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:32:41.460 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-11T22:32:41.461 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:32:41.461 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:32:41.461 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:32:41.461 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17931 2020-03-11T22:32:41.461 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:32:41.462 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:32:41.462 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2020-03-11T22:32:41.462 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-11T22:32:41.462 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:32:41.462 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:32:41.463 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.17931 2020-03-11T22:32:41.463 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-11T22:32:41.463 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2020-03-11T22:32:41.463 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2020-03-11T22:32:41.463 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-11T22:32:41.463 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2020-03-11T22:32:41.464 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent td/divergent-priors 2020-03-11T22:32:41.464 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:57: TEST_divergent: local dir=td/divergent-priors 2020-03-11T22:32:41.464 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:60: TEST_divergent: local dummyfile=/etc/fstab 2020-03-11T22:32:41.464 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:61: TEST_divergent: local dummyfile2=/etc/resolv.conf 2020-03-11T22:32:41.464 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:63: TEST_divergent: local num_osds=3 2020-03-11T22:32:41.465 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: expr 3 - 1 2020-03-11T22:32:41.465 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: seq 0 2 2020-03-11T22:32:41.465 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: local 'osds=0 2020-03-11T22:32:41.465 INFO:tasks.workunit.client.0.smithi059.stderr:1 2020-03-11T22:32:41.465 INFO:tasks.workunit.client.0.smithi059.stderr:2' 2020-03-11T22:32:41.466 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:65: TEST_divergent: run_mon td/divergent-priors a 2020-03-11T22:32:41.466 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2020-03-11T22:32:41.466 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-11T22:32:41.466 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-11T22:32:41.466 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-11T22:32:41.466 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2020-03-11T22:32:41.467 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2020-03-11T22:32:41.750 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-11T22:32:41.750 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:32:41.750 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:32:41.751 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:32:41.751 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:32:41.751 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:32:41.751 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17931/$cluster-$name.asok' 2020-03-11T22:32:41.751 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-11T22:32:41.915 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-11T22:32:41.916 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-11T22:32:41.917 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:32:41.917 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:32:41.917 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-11T22:32:41.917 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-11T22:32:41.917 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:32:41.917 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:32:41.918 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:32:41.918 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:32:41.918 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:32:41.918 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:32:41.918 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17931/ceph-mon.a.asok 2020-03-11T22:32:41.918 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:32:41.918 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.17931/ceph-mon.a.asok config get fsid 2020-03-11T22:32:42.009 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-11T22:32:42.009 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:32:42.009 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:32:42.009 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-11T22:32:42.010 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-11T22:32:42.010 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:32:42.010 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:32:42.010 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:32:42.010 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:32:42.010 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:32:42.010 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:32:42.011 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17931/ceph-mon.a.asok 2020-03-11T22:32:42.011 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:32:42.011 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.17931/ceph-mon.a.asok config get mon_host 2020-03-11T22:32:42.099 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:66: TEST_divergent: run_mgr td/divergent-priors x 2020-03-11T22:32:42.099 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/divergent-priors 2020-03-11T22:32:42.099 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-11T22:32:42.100 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-11T22:32:42.100 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-11T22:32:42.100 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/divergent-priors/x 2020-03-11T22:32:42.100 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-11T22:32:43.000 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-11T22:32:43.001 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:32:43.001 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:32:43.001 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:32:43.001 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:32:43.001 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:32:43.001 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17931/$cluster-$name.asok' 2020-03-11T22:32:43.002 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:32:43.002 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:32:43.031 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2020-03-11T22:32:43.031 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 0 2020-03-11T22:32:43.031 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:32:43.031 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:32:43.031 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-11T22:32:43.032 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:32:43.032 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/0 2020-03-11T22:32:43.032 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=cb7f2e42-1934-4145-a3c3-29bdd93f8adb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:32:43.032 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:32:43.032 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:32:43.032 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:32:43.033 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-11T22:32:43.033 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-11T22:32:43.033 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:32:43.033 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:32:43.034 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:32:43.034 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:32:43.034 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:32:43.034 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:32:43.034 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:32:43.034 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:32:43.034 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:32:43.035 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17931/$cluster-$name.asok' 2020-03-11T22:32:43.035 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' 2020-03-11T22:32:43.035 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:32:43.035 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:32:43.035 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:32:43.035 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:32:43.035 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:32:43.035 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:32:43.036 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:32:43.036 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:32:43.036 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:32:43.036 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:32:43.036 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/0 2020-03-11T22:32:43.036 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:32:43.037 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=3024509c-3f4d-4606-be45-2b0ba42736ad 2020-03-11T22:32:43.037 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 3024509c-3f4d-4606-be45-2b0ba42736ad' 2020-03-11T22:32:43.037 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:32:43.037 INFO:tasks.workunit.client.0.smithi059.stdout:add osd0 3024509c-3f4d-4606-be45-2b0ba42736ad 2020-03-11T22:32:43.050 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQALZ2le+WrZAhAApPnAxxAFTK0MDFUdljJmog== 2020-03-11T22:32:43.050 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQALZ2le+WrZAhAApPnAxxAFTK0MDFUdljJmog=="}' 2020-03-11T22:32:43.051 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 3024509c-3f4d-4606-be45-2b0ba42736ad -i td/divergent-priors/0/new.json 2020-03-11T22:32:43.409 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:32:43.418 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/0/new.json 2020-03-11T22:32:43.419 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=cb7f2e42-1934-4145-a3c3-29bdd93f8adb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQALZ2le+WrZAhAApPnAxxAFTK0MDFUdljJmog== --osd-uuid 3024509c-3f4d-4606-be45-2b0ba42736ad 2020-03-11T22:32:43.449 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:32:43.446+0000 7f539c586ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:32:43.451 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:32:43.448+0000 7f539c586ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:32:43.452 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:32:43.449+0000 7f539c586ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:32:43.453 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:32:43.450+0000 7f539c586ec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:32:43.453 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:32:43.450+0000 7f539c586ec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:32:43.453 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:32:43.450+0000 7f539c586ec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:32:43.557 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:32:43.555+0000 7f539c586ec0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2020-03-11T22:32:46.614 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/0/keyring 2020-03-11T22:32:46.614 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:32:46.615 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-11T22:32:46.615 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:32:46.615 INFO:tasks.workunit.client.0.smithi059.stdout:adding osd0 key to auth repository 2020-03-11T22:32:46.934 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.0 2020-03-11T22:32:46.934 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-11T22:32:46.934 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:32:46.934 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=cb7f2e42-1934-4145-a3c3-29bdd93f8adb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:32:46.934 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:32:46.934 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:32:46.961 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:32:46.959+0000 7fae54f63ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:32:46.974 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:32:46.971+0000 7fae54f63ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:32:46.975 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:32:46.972+0000 7fae54f63ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:32:47.251 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-11T22:32:47.252 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:32:47.252 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:32:47.252 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:32:47.252 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:32:47.252 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:32:47.252 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:32:47.252 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:32:47.253 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:32:47.253 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:32:47.491 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:32:47.488+0000 7fae54f63ec0 -1 Falling back to public interface 2020-03-11T22:32:47.618 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:32:48.223 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:32:48.221+0000 7fae54f63ec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-11T22:32:48.619 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:32:48.619 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:32:48.619 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:32:48.620 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:32:48.620 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:32:48.620 INFO:tasks.workunit.client.0.smithi059.stdout:1 2020-03-11T22:32:49.012 INFO:tasks.workunit.client.0.smithi059.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/18313,v1:127.0.0.1:6803/18313] [v2:127.0.0.1:6804/18313,v1:127.0.0.1:6805/18313] exists,up 3024509c-3f4d-4606-be45-2b0ba42736ad 2020-03-11T22:32:49.012 INFO:tasks.workunit.client.0.smithi059.stdout:add osd1 85494619-d02f-47a9-a790-661f9ba109cb 2020-03-11T22:32:49.012 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:32:49.012 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:32:49.012 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:32:49.013 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2020-03-11T22:32:49.013 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 1 2020-03-11T22:32:49.013 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:32:49.013 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:32:49.013 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-11T22:32:49.013 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:32:49.013 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:32:49.013 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=cb7f2e42-1934-4145-a3c3-29bdd93f8adb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:32:49.014 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:32:49.014 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:32:49.014 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:32:49.014 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:32:49.014 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:32:49.014 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:32:49.014 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:32:49.015 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:32:49.015 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:32:49.015 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:32:49.015 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:32:49.015 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:32:49.015 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:32:49.015 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:32:49.015 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17931/$cluster-$name.asok' 2020-03-11T22:32:49.016 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' 2020-03-11T22:32:49.016 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:32:49.016 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:32:49.016 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:32:49.016 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:32:49.016 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:32:49.016 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:32:49.017 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:32:49.017 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:32:49.017 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:32:49.017 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:32:49.017 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:32:49.017 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:32:49.018 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=85494619-d02f-47a9-a790-661f9ba109cb 2020-03-11T22:32:49.018 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 85494619-d02f-47a9-a790-661f9ba109cb' 2020-03-11T22:32:49.018 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:32:49.031 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQARZ2leRnOtARAAAN+ZqFYvB5el+YgK1nbgIQ== 2020-03-11T22:32:49.031 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQARZ2leRnOtARAAAN+ZqFYvB5el+YgK1nbgIQ=="}' 2020-03-11T22:32:49.031 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 85494619-d02f-47a9-a790-661f9ba109cb -i td/divergent-priors/1/new.json 2020-03-11T22:32:49.620 INFO:tasks.workunit.client.0.smithi059.stdout:1 2020-03-11T22:32:49.630 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/1/new.json 2020-03-11T22:32:49.631 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=cb7f2e42-1934-4145-a3c3-29bdd93f8adb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQARZ2leRnOtARAAAN+ZqFYvB5el+YgK1nbgIQ== --osd-uuid 85494619-d02f-47a9-a790-661f9ba109cb 2020-03-11T22:32:49.661 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:32:49.658+0000 7f803bdf6ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:32:49.701 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:32:49.698+0000 7f803bdf6ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:32:49.702 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:32:49.700+0000 7f803bdf6ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:32:49.703 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:32:49.700+0000 7f803bdf6ec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:32:49.703 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:32:49.700+0000 7f803bdf6ec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:32:49.704 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:32:49.700+0000 7f803bdf6ec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:32:49.772 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:32:49.769+0000 7f803bdf6ec0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2020-03-11T22:32:52.837 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/1/keyring 2020-03-11T22:32:52.837 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:32:52.838 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-03-11T22:32:52.839 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:32:52.839 INFO:tasks.workunit.client.0.smithi059.stdout:adding osd1 key to auth repository 2020-03-11T22:32:53.329 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.1 2020-03-11T22:32:53.329 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-03-11T22:32:53.330 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=cb7f2e42-1934-4145-a3c3-29bdd93f8adb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:32:53.330 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:32:53.330 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:32:53.330 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:32:53.357 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:32:53.354+0000 7f913adafec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:32:53.359 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:32:53.356+0000 7f913adafec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:32:53.361 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:32:53.357+0000 7f913adafec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:32:53.697 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-03-11T22:32:53.697 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:32:53.697 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:32:53.698 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:32:53.698 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:32:53.698 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:32:53.698 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:32:53.698 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:32:53.698 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:32:53.698 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:32:53.875 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:32:53.872+0000 7f913adafec0 -1 Falling back to public interface 2020-03-11T22:32:54.062 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:32:54.595 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:32:54.592+0000 7f913adafec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-11T22:32:55.064 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:32:55.064 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:32:55.064 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:32:55.064 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:32:55.064 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:32:55.065 INFO:tasks.workunit.client.0.smithi059.stdout:1 2020-03-11T22:32:55.469 INFO:tasks.workunit.client.0.smithi059.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/18723,v1:127.0.0.1:6811/18723] [v2:127.0.0.1:6812/18723,v1:127.0.0.1:6813/18723] exists,up 85494619-d02f-47a9-a790-661f9ba109cb 2020-03-11T22:32:55.469 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:32:55.469 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:32:55.469 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:32:55.470 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2020-03-11T22:32:55.470 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 2 2020-03-11T22:32:55.470 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:32:55.470 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:32:55.471 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-03-11T22:32:55.471 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:32:55.471 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/2 2020-03-11T22:32:55.471 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=cb7f2e42-1934-4145-a3c3-29bdd93f8adb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:32:55.471 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:32:55.471 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:32:55.472 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:32:55.472 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-11T22:32:55.472 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-11T22:32:55.472 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:32:55.472 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:32:55.472 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:32:55.473 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:32:55.473 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:32:55.473 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:32:55.473 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:32:55.473 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:32:55.473 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:32:55.473 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17931/$cluster-$name.asok' 2020-03-11T22:32:55.474 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' 2020-03-11T22:32:55.474 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:32:55.474 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:32:55.474 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:32:55.474 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:32:55.474 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:32:55.474 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:32:55.475 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:32:55.475 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:32:55.475 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:32:55.475 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:32:55.475 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/2 2020-03-11T22:32:55.476 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:32:55.476 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=9a5ddff4-d2b7-43cd-b7c4-5b04744a990c 2020-03-11T22:32:55.476 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 9a5ddff4-d2b7-43cd-b7c4-5b04744a990c' 2020-03-11T22:32:55.476 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:32:55.477 INFO:tasks.workunit.client.0.smithi059.stdout:add osd2 9a5ddff4-d2b7-43cd-b7c4-5b04744a990c 2020-03-11T22:32:55.490 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQAXZ2le2tsNHRAAib3EoxUZsoa1rTud3qSoCQ== 2020-03-11T22:32:55.490 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQAXZ2le2tsNHRAAib3EoxUZsoa1rTud3qSoCQ=="}' 2020-03-11T22:32:55.490 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 9a5ddff4-d2b7-43cd-b7c4-5b04744a990c -i td/divergent-priors/2/new.json 2020-03-11T22:32:55.963 INFO:tasks.workunit.client.0.smithi059.stdout:2 2020-03-11T22:32:55.973 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/2/new.json 2020-03-11T22:32:55.974 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=cb7f2e42-1934-4145-a3c3-29bdd93f8adb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAXZ2le2tsNHRAAib3EoxUZsoa1rTud3qSoCQ== --osd-uuid 9a5ddff4-d2b7-43cd-b7c4-5b04744a990c 2020-03-11T22:32:56.004 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:32:56.001+0000 7f2b8f7f9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:32:56.019 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:32:56.016+0000 7f2b8f7f9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:32:56.020 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:32:56.017+0000 7f2b8f7f9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:32:56.020 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:32:56.017+0000 7f2b8f7f9ec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:32:56.020 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:32:56.017+0000 7f2b8f7f9ec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:32:56.021 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:32:56.017+0000 7f2b8f7f9ec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:32:56.085 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:32:56.083+0000 7f2b8f7f9ec0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2020-03-11T22:32:59.128 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/2/keyring 2020-03-11T22:32:59.128 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:32:59.130 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-03-11T22:32:59.130 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:32:59.130 INFO:tasks.workunit.client.0.smithi059.stdout:adding osd2 key to auth repository 2020-03-11T22:32:59.616 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-03-11T22:32:59.616 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=cb7f2e42-1934-4145-a3c3-29bdd93f8adb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:32:59.617 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:32:59.617 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:32:59.617 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:32:59.617 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.2 2020-03-11T22:32:59.644 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:32:59.641+0000 7f9baac41ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:32:59.645 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:32:59.643+0000 7f9baac41ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:32:59.646 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:32:59.644+0000 7f9baac41ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:32:59.983 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-03-11T22:32:59.983 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:32:59.984 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:32:59.984 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:32:59.984 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:32:59.984 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:32:59.984 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:32:59.984 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:32:59.985 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:32:59.985 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:33:00.163 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:33:00.160+0000 7f9baac41ec0 -1 Falling back to public interface 2020-03-11T22:33:00.371 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:33:00.936 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:33:00.933+0000 7f9baac41ec0 -1 osd.2 0 log_to_monitors {default=true} 2020-03-11T22:33:01.373 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:33:01.373 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:33:01.374 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:33:01.374 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:33:01.374 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:33:01.374 INFO:tasks.workunit.client.0.smithi059.stdout:1 2020-03-11T22:33:01.771 INFO:tasks.workunit.client.0.smithi059.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/19078,v1:127.0.0.1:6819/19078] [v2:127.0.0.1:6820/19078,v1:127.0.0.1:6821/19078] exists,up 9a5ddff4-d2b7-43cd-b7c4-5b04744a990c 2020-03-11T22:33:01.771 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:33:01.771 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:33:01.772 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:33:01.772 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:72: TEST_divergent: ceph osd set noout 2020-03-11T22:33:02.546 INFO:tasks.workunit.client.0.smithi059.stderr:noout is set 2020-03-11T22:33:02.554 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:73: TEST_divergent: ceph osd set noin 2020-03-11T22:33:03.173 INFO:tasks.workunit.client.0.smithi059.stderr:noin is set 2020-03-11T22:33:03.181 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:74: TEST_divergent: ceph osd set nodown 2020-03-11T22:33:04.025 INFO:tasks.workunit.client.0.smithi059.stderr:nodown is set 2020-03-11T22:33:04.034 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:75: TEST_divergent: create_pool test 1 1 2020-03-11T22:33:04.034 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test 1 1 2020-03-11T22:33:04.596 INFO:tasks.workunit.client.0.smithi059.stderr:pool 'test' already exists 2020-03-11T22:33:04.604 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:33:05.605 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:76: TEST_divergent: ceph osd pool set test size 3 2020-03-11T22:33:06.245 INFO:tasks.workunit.client.0.smithi059.stderr:set pool 1 size to 3 2020-03-11T22:33:06.253 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:77: TEST_divergent: ceph osd pool set test min_size 2 2020-03-11T22:33:07.046 INFO:tasks.workunit.client.0.smithi059.stderr:set pool 1 min_size to 2 2020-03-11T22:33:07.054 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:79: TEST_divergent: flush_pg_stats 2020-03-11T22:33:07.054 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:33:07.055 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:33:07.421 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:33:07.421 INFO:tasks.workunit.client.0.smithi059.stderr:1 2020-03-11T22:33:07.421 INFO:tasks.workunit.client.0.smithi059.stderr:2' 2020-03-11T22:33:07.421 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:33:07.421 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:33:07.422 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:33:07.558 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836486 2020-03-11T22:33:07.558 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486' 2020-03-11T22:33:07.558 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:33:07.558 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:33:07.695 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672965 2020-03-11T22:33:07.696 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486 1-42949672965' 2020-03-11T22:33:07.696 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:33:07.696 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:33:07.833 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509444 2020-03-11T22:33:07.833 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486 1-42949672965 2-64424509444' 2020-03-11T22:33:07.833 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:33:07.834 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836486 2020-03-11T22:33:07.834 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:33:07.834 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:33:07.836 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836486 2020-03-11T22:33:07.836 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:33:07.837 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836486 2020-03-11T22:33:07.837 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2020-03-11T22:33:07.837 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.0 seq 21474836486 2020-03-11T22:33:07.837 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:33:08.202 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836484 -lt 21474836486 2020-03-11T22:33:08.202 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:33:09.203 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:33:09.204 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:33:09.578 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836486 -lt 21474836486 2020-03-11T22:33:09.578 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:33:09.579 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672965 2020-03-11T22:33:09.579 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:33:09.580 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:33:09.581 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672965 2020-03-11T22:33:09.581 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:33:09.582 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672965 2020-03-11T22:33:09.582 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2020-03-11T22:33:09.583 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.1 seq 42949672965 2020-03-11T22:33:09.584 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:33:09.967 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672965 -lt 42949672965 2020-03-11T22:33:09.968 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:33:09.968 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509444 2020-03-11T22:33:09.969 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:33:09.969 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:33:09.970 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509444 2020-03-11T22:33:09.971 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:33:09.972 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509444 2020-03-11T22:33:09.972 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2020-03-11T22:33:09.972 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.2 seq 64424509444 2020-03-11T22:33:09.973 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:33:10.338 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509444 -lt 64424509444 2020-03-11T22:33:10.338 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:80: TEST_divergent: wait_for_clean 2020-03-11T22:33:10.339 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:33:10.339 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:33:10.339 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:33:10.339 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:33:10.340 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:33:10.340 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:33:10.340 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:33:10.341 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:33:10.341 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:33:10.341 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:33:10.432 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:33:10.432 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:33:10.432 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:33:10.432 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:33:10.433 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:33:10.795 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:33:10.795 INFO:tasks.workunit.client.0.smithi059.stderr:1 2020-03-11T22:33:10.796 INFO:tasks.workunit.client.0.smithi059.stderr:2' 2020-03-11T22:33:10.796 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:33:10.796 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:33:10.796 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:33:10.982 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836489 2020-03-11T22:33:10.983 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836489' 2020-03-11T22:33:10.983 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:33:10.983 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:33:11.133 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672968 2020-03-11T22:33:11.133 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836489 1-42949672968' 2020-03-11T22:33:11.134 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:33:11.134 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:33:11.270 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509446 2020-03-11T22:33:11.270 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836489 1-42949672968 2-64424509446' 2020-03-11T22:33:11.271 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:33:11.271 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836489 2020-03-11T22:33:11.271 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:33:11.272 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:33:11.273 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836489 2020-03-11T22:33:11.274 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:33:11.274 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836489 2020-03-11T22:33:11.275 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2020-03-11T22:33:11.275 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.0 seq 21474836489 2020-03-11T22:33:11.276 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:33:11.639 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836489 -lt 21474836489 2020-03-11T22:33:11.639 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:33:11.640 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672968 2020-03-11T22:33:11.640 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:33:11.641 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:33:11.642 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672968 2020-03-11T22:33:11.642 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:33:11.643 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672968 2020-03-11T22:33:11.643 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2020-03-11T22:33:11.643 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.1 seq 42949672968 2020-03-11T22:33:11.644 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:33:12.045 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672966 -lt 42949672968 2020-03-11T22:33:12.045 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:33:13.046 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:33:13.047 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:33:14.150 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672968 -lt 42949672968 2020-03-11T22:33:14.151 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:33:14.151 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509446 2020-03-11T22:33:14.151 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:33:14.152 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:33:14.153 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509446 2020-03-11T22:33:14.154 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:33:14.154 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509446 2020-03-11T22:33:14.155 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2020-03-11T22:33:14.155 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.2 seq 64424509446 2020-03-11T22:33:14.155 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:33:14.517 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509447 -lt 64424509446 2020-03-11T22:33:14.518 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:33:14.518 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:33:14.518 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:33:14.992 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:33:14.993 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:33:14.994 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:33:14.994 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:33:14.994 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:33:14.994 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:33:14.994 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:33:14.994 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:33:15.350 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:33:15.351 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:33:15.351 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:33:15.351 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:33:15.824 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:33:15.824 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:33:15.824 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:33:15.825 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: ceph pg dump pgs --format=json 2020-03-11T22:33:15.825 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: jq '.pg_stats[0].up_primary' 2020-03-11T22:33:16.166 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2020-03-11T22:33:16.174 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: local divergent=1 2020-03-11T22:33:16.175 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:84: TEST_divergent: echo 'primary and soon to be divergent is 1' 2020-03-11T22:33:16.175 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:85: TEST_divergent: ceph pg dump pgs 2020-03-11T22:33:16.175 INFO:tasks.workunit.client.0.smithi059.stdout:primary and soon to be divergent is 1 2020-03-11T22:33:16.517 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:33:16.518 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:33:05.147892+0000 0'0 26:11 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:33:04.364850+0000 0'0 2020-03-11T22:33:04.364850+0000 0 2020-03-11T22:33:16.518 INFO:tasks.workunit.client.0.smithi059.stdout: 2020-03-11T22:33:16.518 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:33:16.519 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2020-03-11T22:33:16.528 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:86: TEST_divergent: local non_divergent= 2020-03-11T22:33:16.528 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2020-03-11T22:33:16.528 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 0 = 1 ']' 2020-03-11T22:33:16.528 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:92: TEST_divergent: non_divergent=' 0' 2020-03-11T22:33:16.528 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2020-03-11T22:33:16.528 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 1 = 1 ']' 2020-03-11T22:33:16.529 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:90: TEST_divergent: continue 2020-03-11T22:33:16.529 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2020-03-11T22:33:16.529 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 2 = 1 ']' 2020-03-11T22:33:16.529 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:92: TEST_divergent: non_divergent=' 0 2' 2020-03-11T22:33:16.529 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:95: TEST_divergent: echo 'writing initial objects' 2020-03-11T22:33:16.529 INFO:tasks.workunit.client.0.smithi059.stdout:writing initial objects 2020-03-11T22:33:16.530 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: seq 1 100 2020-03-11T22:33:16.530 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:16.530 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_1 /etc/fstab 2020-03-11T22:33:16.576 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:16.576 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_2 /etc/fstab 2020-03-11T22:33:16.617 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:16.653 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_3 /etc/fstab 2020-03-11T22:33:16.654 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:16.654 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_4 /etc/fstab 2020-03-11T22:33:16.693 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:16.693 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_5 /etc/fstab 2020-03-11T22:33:16.737 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:16.737 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_6 /etc/fstab 2020-03-11T22:33:16.776 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:16.776 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_7 /etc/fstab 2020-03-11T22:33:16.859 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:16.860 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_8 /etc/fstab 2020-03-11T22:33:16.900 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:16.900 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_9 /etc/fstab 2020-03-11T22:33:16.938 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:16.938 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_10 /etc/fstab 2020-03-11T22:33:17.025 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:17.025 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_11 /etc/fstab 2020-03-11T22:33:17.066 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:17.067 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_12 /etc/fstab 2020-03-11T22:33:17.226 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:17.231 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_13 /etc/fstab 2020-03-11T22:33:17.268 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:17.268 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_14 /etc/fstab 2020-03-11T22:33:17.308 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:17.308 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_15 /etc/fstab 2020-03-11T22:33:17.346 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:17.346 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_16 /etc/fstab 2020-03-11T22:33:17.393 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:17.393 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_17 /etc/fstab 2020-03-11T22:33:17.433 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:17.433 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_18 /etc/fstab 2020-03-11T22:33:17.471 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:17.471 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_19 /etc/fstab 2020-03-11T22:33:17.551 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:17.551 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_20 /etc/fstab 2020-03-11T22:33:17.593 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:17.593 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_21 /etc/fstab 2020-03-11T22:33:17.633 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:17.633 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_22 /etc/fstab 2020-03-11T22:33:17.728 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:17.728 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_23 /etc/fstab 2020-03-11T22:33:17.768 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:17.804 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_24 /etc/fstab 2020-03-11T22:33:17.852 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:17.852 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_25 /etc/fstab 2020-03-11T22:33:17.893 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:17.894 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_26 /etc/fstab 2020-03-11T22:33:17.958 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:17.958 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_27 /etc/fstab 2020-03-11T22:33:17.996 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:18.005 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_28 /etc/fstab 2020-03-11T22:33:18.035 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:18.035 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_29 /etc/fstab 2020-03-11T22:33:18.075 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:18.075 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_30 /etc/fstab 2020-03-11T22:33:18.113 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:18.113 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_31 /etc/fstab 2020-03-11T22:33:18.194 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:18.194 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_32 /etc/fstab 2020-03-11T22:33:18.243 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:18.243 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_33 /etc/fstab 2020-03-11T22:33:18.352 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:18.353 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_34 /etc/fstab 2020-03-11T22:33:18.393 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:18.393 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_35 /etc/fstab 2020-03-11T22:33:18.433 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:18.434 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_36 /etc/fstab 2020-03-11T22:33:18.536 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:18.550 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_37 /etc/fstab 2020-03-11T22:33:18.577 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:18.577 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_38 /etc/fstab 2020-03-11T22:33:18.617 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:18.617 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_39 /etc/fstab 2020-03-11T22:33:18.686 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:18.686 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_40 /etc/fstab 2020-03-11T22:33:18.727 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:18.727 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_41 /etc/fstab 2020-03-11T22:33:18.795 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:18.795 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_42 /etc/fstab 2020-03-11T22:33:18.842 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:18.842 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_43 /etc/fstab 2020-03-11T22:33:18.920 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:18.920 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_44 /etc/fstab 2020-03-11T22:33:19.068 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:19.068 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_45 /etc/fstab 2020-03-11T22:33:19.234 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:19.235 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_46 /etc/fstab 2020-03-11T22:33:19.272 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:19.272 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_47 /etc/fstab 2020-03-11T22:33:19.315 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:19.315 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_48 /etc/fstab 2020-03-11T22:33:19.426 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:19.426 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_49 /etc/fstab 2020-03-11T22:33:19.463 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:19.464 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_50 /etc/fstab 2020-03-11T22:33:19.502 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:19.502 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_51 /etc/fstab 2020-03-11T22:33:19.542 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:19.543 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_52 /etc/fstab 2020-03-11T22:33:19.588 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:19.588 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_53 /etc/fstab 2020-03-11T22:33:19.636 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:19.636 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_54 /etc/fstab 2020-03-11T22:33:19.686 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:19.686 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_55 /etc/fstab 2020-03-11T22:33:19.764 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:19.764 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_56 /etc/fstab 2020-03-11T22:33:19.805 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:19.805 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_57 /etc/fstab 2020-03-11T22:33:19.889 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:19.889 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_58 /etc/fstab 2020-03-11T22:33:19.930 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:19.930 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_59 /etc/fstab 2020-03-11T22:33:20.022 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:20.022 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_60 /etc/fstab 2020-03-11T22:33:20.063 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:20.063 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_61 /etc/fstab 2020-03-11T22:33:20.102 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:20.103 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_62 /etc/fstab 2020-03-11T22:33:20.143 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:20.143 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_63 /etc/fstab 2020-03-11T22:33:20.180 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:20.180 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_64 /etc/fstab 2020-03-11T22:33:20.219 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:20.219 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_65 /etc/fstab 2020-03-11T22:33:20.264 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:20.264 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_66 /etc/fstab 2020-03-11T22:33:20.348 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:20.348 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_67 /etc/fstab 2020-03-11T22:33:20.388 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:20.389 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_68 /etc/fstab 2020-03-11T22:33:20.473 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:20.473 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_69 /etc/fstab 2020-03-11T22:33:20.536 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:20.536 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_70 /etc/fstab 2020-03-11T22:33:20.581 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:20.581 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_71 /etc/fstab 2020-03-11T22:33:20.673 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:20.825 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_72 /etc/fstab 2020-03-11T22:33:20.826 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:20.826 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_73 /etc/fstab 2020-03-11T22:33:20.826 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:20.826 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_74 /etc/fstab 2020-03-11T22:33:20.827 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:20.827 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_75 /etc/fstab 2020-03-11T22:33:20.859 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:20.860 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_76 /etc/fstab 2020-03-11T22:33:20.897 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:20.897 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_77 /etc/fstab 2020-03-11T22:33:20.936 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:20.937 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_78 /etc/fstab 2020-03-11T22:33:21.015 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:21.015 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_79 /etc/fstab 2020-03-11T22:33:21.055 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:21.056 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_80 /etc/fstab 2020-03-11T22:33:21.199 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:21.199 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_81 /etc/fstab 2020-03-11T22:33:21.255 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:21.256 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_82 /etc/fstab 2020-03-11T22:33:21.400 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:21.400 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_83 /etc/fstab 2020-03-11T22:33:21.458 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:21.460 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_84 /etc/fstab 2020-03-11T22:33:21.501 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:21.501 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_85 /etc/fstab 2020-03-11T22:33:21.539 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:21.539 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_86 /etc/fstab 2020-03-11T22:33:21.578 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:21.578 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_87 /etc/fstab 2020-03-11T22:33:21.618 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:21.619 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_88 /etc/fstab 2020-03-11T22:33:21.708 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:21.709 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_89 /etc/fstab 2020-03-11T22:33:21.747 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:21.748 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_90 /etc/fstab 2020-03-11T22:33:21.825 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:21.825 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_91 /etc/fstab 2020-03-11T22:33:21.864 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:21.864 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_92 /etc/fstab 2020-03-11T22:33:21.968 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:21.968 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_93 /etc/fstab 2020-03-11T22:33:22.051 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:22.051 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_94 /etc/fstab 2020-03-11T22:33:22.126 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:22.126 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_95 /etc/fstab 2020-03-11T22:33:22.172 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:22.172 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_96 /etc/fstab 2020-03-11T22:33:22.231 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:22.231 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_97 /etc/fstab 2020-03-11T22:33:22.270 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:22.270 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_98 /etc/fstab 2020-03-11T22:33:22.310 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:22.310 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_99 /etc/fstab 2020-03-11T22:33:22.347 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:33:22.347 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_100 /etc/fstab 2020-03-11T22:33:22.387 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:102: TEST_divergent: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:33:22.387 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:102: TEST_divergent: wait_for_clean 2020-03-11T22:33:22.387 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:33:22.388 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:33:22.388 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:33:22.388 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:33:22.389 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:33:22.389 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:33:22.389 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:33:22.389 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:33:22.389 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:33:22.389 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:33:22.445 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:33:22.446 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:33:22.446 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:33:22.446 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:33:22.446 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:33:22.810 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:33:22.810 INFO:tasks.workunit.client.0.smithi059.stderr:1 2020-03-11T22:33:22.810 INFO:tasks.workunit.client.0.smithi059.stderr:2' 2020-03-11T22:33:22.810 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:33:22.811 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:33:22.811 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:33:22.946 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836493 2020-03-11T22:33:22.946 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493' 2020-03-11T22:33:22.947 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:33:22.947 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:33:23.084 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672972 2020-03-11T22:33:23.084 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-42949672972' 2020-03-11T22:33:23.085 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:33:23.085 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:33:23.221 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509451 2020-03-11T22:33:23.221 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-42949672972 2-64424509451' 2020-03-11T22:33:23.222 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:33:23.222 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836493 2020-03-11T22:33:23.222 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:33:23.223 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:33:23.224 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836493 2020-03-11T22:33:23.224 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:33:23.225 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836493 2020-03-11T22:33:23.225 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2020-03-11T22:33:23.226 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.0 seq 21474836493 2020-03-11T22:33:23.226 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:33:23.590 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836493 -lt 21474836493 2020-03-11T22:33:23.590 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:33:23.591 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672972 2020-03-11T22:33:23.591 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:33:23.592 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:33:23.593 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672972 2020-03-11T22:33:23.593 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:33:23.594 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672972 2020-03-11T22:33:23.594 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2020-03-11T22:33:23.594 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.1 seq 42949672972 2020-03-11T22:33:23.595 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:33:23.966 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672970 -lt 42949672972 2020-03-11T22:33:23.966 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:33:24.967 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:33:24.968 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:33:25.355 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672972 -lt 42949672972 2020-03-11T22:33:25.355 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:33:25.356 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509451 2020-03-11T22:33:25.356 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:33:25.357 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:33:25.358 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509451 2020-03-11T22:33:25.358 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:33:25.359 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509451 2020-03-11T22:33:25.359 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2020-03-11T22:33:25.360 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.2 seq 64424509451 2020-03-11T22:33:25.360 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:33:25.724 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509451 -lt 64424509451 2020-03-11T22:33:25.725 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:33:25.725 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:33:25.726 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:33:26.205 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:33:26.206 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:33:26.206 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:33:26.207 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:33:26.207 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:33:26.207 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:33:26.207 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:33:26.207 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:33:26.558 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:33:26.559 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:33:26.560 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:33:26.560 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:33:27.035 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:33:27.035 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:33:27.035 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:33:27.036 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:104: TEST_divergent: get_pg test existing_1 2020-03-11T22:33:27.036 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1030: get_pg: local poolname=test 2020-03-11T22:33:27.036 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local objectname=existing_1 2020-03-11T22:33:27.036 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: ceph --format json osd map test existing_1 2020-03-11T22:33:27.037 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: jq -r .pgid 2020-03-11T22:33:27.413 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:104: TEST_divergent: local pgid=1.0 2020-03-11T22:33:27.413 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:107: TEST_divergent: echo 'blackholing osds 0 2' 2020-03-11T22:33:27.413 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:108: TEST_divergent: ceph pg dump pgs 2020-03-11T22:33:27.413 INFO:tasks.workunit.client.0.smithi059.stdout:blackholing osds 0 2 2020-03-11T22:33:27.758 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:33:27.758 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 100 0 0 0 0 48300 0 0 16 16 active+clean 2020-03-11T22:33:05.147892+0000 29'100 29:111 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:33:04.364850+0000 0'0 2020-03-11T22:33:04.364850+0000 0 2020-03-11T22:33:27.758 INFO:tasks.workunit.client.0.smithi059.stdout: 2020-03-11T22:33:27.759 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:33:27.760 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2020-03-11T22:33:27.768 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:109: TEST_divergent: for i in $non_divergent 2020-03-11T22:33:27.769 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: get_asok_path osd.0 2020-03-11T22:33:27.769 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-11T22:33:27.769 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-11T22:33:27.770 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:33:27.770 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:33:27.770 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:33:27.770 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17931/ceph-osd.0.asok 2020-03-11T22:33:27.770 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: CEPH_ARGS= 2020-03-11T22:33:27.771 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.17931/ceph-osd.0.asok config set objectstore_blackhole 1 2020-03-11T22:33:27.861 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2020-03-11T22:33:27.861 INFO:tasks.workunit.client.0.smithi059.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-03-11T22:33:27.861 INFO:tasks.workunit.client.0.smithi059.stdout:} 2020-03-11T22:33:27.868 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:109: TEST_divergent: for i in $non_divergent 2020-03-11T22:33:27.870 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: get_asok_path osd.2 2020-03-11T22:33:27.870 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-03-11T22:33:27.871 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-03-11T22:33:27.871 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:33:27.871 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:33:27.871 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:33:27.871 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17931/ceph-osd.2.asok 2020-03-11T22:33:27.871 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: CEPH_ARGS= 2020-03-11T22:33:27.871 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.17931/ceph-osd.2.asok config set objectstore_blackhole 1 2020-03-11T22:33:27.961 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2020-03-11T22:33:27.962 INFO:tasks.workunit.client.0.smithi059.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-03-11T22:33:27.962 INFO:tasks.workunit.client.0.smithi059.stdout:} 2020-03-11T22:33:27.968 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:114: TEST_divergent: local case5=100 2020-03-11T22:33:27.969 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:115: TEST_divergent: expr 100 - 1 2020-03-11T22:33:27.973 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:115: TEST_divergent: local case3=99 2020-03-11T22:33:27.973 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:117: TEST_divergent: echo 'writing divergent object' 2020-03-11T22:33:27.973 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:119: TEST_divergent: echo 'create missing divergent object' 2020-03-11T22:33:27.973 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:120: TEST_divergent: inject_eio rep data test existing_99 td/divergent-priors 0 2020-03-11T22:33:27.973 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:118: TEST_divergent: rados -p test put existing_100 /etc/fstab 2020-03-11T22:33:27.974 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: inject_eio: local pooltype=rep 2020-03-11T22:33:27.974 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: inject_eio: shift 2020-03-11T22:33:27.974 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: inject_eio: local which=data 2020-03-11T22:33:27.974 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: inject_eio: shift 2020-03-11T22:33:27.974 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: inject_eio: local poolname=test 2020-03-11T22:33:27.974 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: inject_eio: shift 2020-03-11T22:33:27.975 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: local objname=existing_99 2020-03-11T22:33:27.976 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: shift 2020-03-11T22:33:27.976 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: local dir=td/divergent-priors 2020-03-11T22:33:27.976 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: shift 2020-03-11T22:33:27.976 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: local shard_id=0 2020-03-11T22:33:27.976 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: shift 2020-03-11T22:33:27.977 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2020-03-11T22:33:27.977 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: get_osds test existing_99 2020-03-11T22:33:27.977 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:958: get_osds: local poolname=test 2020-03-11T22:33:27.977 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local objectname=existing_99 2020-03-11T22:33:27.977 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: get_osds: ceph --format json osd map test existing_99 2020-03-11T22:33:27.977 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: get_osds: jq '.acting | .[]' 2020-03-11T22:33:27.978 INFO:tasks.workunit.client.0.smithi059.stdout:writing divergent object 2020-03-11T22:33:27.978 INFO:tasks.workunit.client.0.smithi059.stdout:create missing divergent object 2020-03-11T22:33:28.346 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: get_osds: local 'osds=1 2020-03-11T22:33:28.346 INFO:tasks.workunit.client.0.smithi059.stderr:0 2020-03-11T22:33:28.346 INFO:tasks.workunit.client.0.smithi059.stderr:2' 2020-03-11T22:33:28.346 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:964: get_osds: echo 1 0 2 2020-03-11T22:33:28.347 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local -a initial_osds 2020-03-11T22:33:28.347 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: local osd_id=1 2020-03-11T22:33:28.347 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: '[' rep '!=' ec ']' 2020-03-11T22:33:28.347 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shard_id= 2020-03-11T22:33:28.348 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: cat td/divergent-priors/1/type 2020-03-11T22:33:28.348 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: type=bluestore 2020-03-11T22:33:28.348 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2020-03-11T22:33:28.348 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1098: set_config: local daemon=osd 2020-03-11T22:33:28.349 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local id=1 2020-03-11T22:33:28.349 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local config=bluestore_debug_inject_read_err 2020-03-11T22:33:28.349 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local value=true 2020-03-11T22:33:28.350 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1105: set_config: jq 'has("success")' 2020-03-11T22:33:28.350 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1105: set_config: get_asok_path osd.1 2020-03-11T22:33:28.350 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:33:28.350 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:33:28.351 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:33:28.351 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:33:28.351 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:33:28.351 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17931/ceph-osd.1.asok 2020-03-11T22:33:28.351 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1105: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.17931/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2020-03-11T22:33:28.449 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1105: set_config: test true == true 2020-03-11T22:33:28.450 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: local loop=0 2020-03-11T22:33:28.451 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2248: inject_eio: grep -q Invalid 2020-03-11T22:33:28.451 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: get_asok_path osd.1 2020-03-11T22:33:28.451 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:33:28.451 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:33:28.452 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:33:28.452 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:33:28.452 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:33:28.452 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17931/ceph-osd.1.asok 2020-03-11T22:33:28.452 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: CEPH_ARGS= 2020-03-11T22:33:28.452 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: ceph --admin-daemon /tmp/ceph-asok.17931/ceph-osd.1.asok injectdataerr test existing_99 2020-03-11T22:33:28.554 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:122: TEST_divergent: sleep 10 2020-03-11T22:33:28.554 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:121: TEST_divergent: rados -p test get existing_99 td/divergent-priors/existing 2020-03-11T22:33:38.555 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:123: TEST_divergent: killall -9 rados 2020-03-11T22:33:38.575 INFO:tasks.workunit.client.0.smithi059.stderr:rados: no process found 2020-03-11T22:33:38.575 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:126: TEST_divergent: echo 'killing all the osds' 2020-03-11T22:33:38.576 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:127: TEST_divergent: ceph pg dump pgs 2020-03-11T22:33:38.576 INFO:tasks.workunit.client.0.smithi059.stdout:killing all the osds 2020-03-11T22:33:38.925 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:33:38.925 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 100 0 0 0 0 48300 0 0 17 17 active+clean 2020-03-11T22:33:28.807245+0000 29'101 29:120 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:33:04.364850+0000 0'0 2020-03-11T22:33:04.364850+0000 0 2020-03-11T22:33:38.925 INFO:tasks.workunit.client.0.smithi059.stdout: 2020-03-11T22:33:38.925 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:33:38.926 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2020-03-11T22:33:38.934 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:128: TEST_divergent: kill_daemons td/divergent-priors KILL osd 2020-03-11T22:33:38.935 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:33:38.936 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:33:38.936 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:33:38.936 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:33:38.936 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:33:39.042 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:33:39.043 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2020-03-11T22:33:39.043 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.0 2020-03-11T22:33:39.591 INFO:tasks.workunit.client.0.smithi059.stderr:osd.0 is already down. 2020-03-11T22:33:39.601 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2020-03-11T22:33:39.601 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.1 2020-03-11T22:33:40.144 INFO:tasks.workunit.client.0.smithi059.stderr:osd.1 is already down. 2020-03-11T22:33:40.152 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2020-03-11T22:33:40.153 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.2 2020-03-11T22:33:40.691 INFO:tasks.workunit.client.0.smithi059.stderr:osd.2 is already down. 2020-03-11T22:33:40.698 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:133: TEST_divergent: for i in $non_divergent 2020-03-11T22:33:40.699 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:135: TEST_divergent: ceph osd out osd.0 2020-03-11T22:33:41.210 INFO:tasks.workunit.client.0.smithi059.stderr:osd.0 is already out. 2020-03-11T22:33:41.218 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:133: TEST_divergent: for i in $non_divergent 2020-03-11T22:33:41.218 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:135: TEST_divergent: ceph osd out osd.2 2020-03-11T22:33:41.726 INFO:tasks.workunit.client.0.smithi059.stderr:osd.2 is already out. 2020-03-11T22:33:41.734 INFO:tasks.workunit.client.0.smithi059.stdout:bringing up non_divergent 0 2 2020-03-11T22:33:41.735 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:139: TEST_divergent: echo 'bringing up non_divergent 0 2' 2020-03-11T22:33:41.735 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:140: TEST_divergent: ceph pg dump pgs 2020-03-11T22:33:42.076 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:33:42.076 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 100 0 0 0 0 48300 0 0 17 17 stale+active+clean 2020-03-11T22:33:28.807245+0000 29'101 29:120 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:33:04.364850+0000 0'0 2020-03-11T22:33:04.364850+0000 0 2020-03-11T22:33:42.076 INFO:tasks.workunit.client.0.smithi059.stdout: 2020-03-11T22:33:42.076 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:33:42.078 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2020-03-11T22:33:42.089 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:141: TEST_divergent: for i in $non_divergent 2020-03-11T22:33:42.089 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:143: TEST_divergent: activate_osd td/divergent-priors 0 2020-03-11T22:33:42.089 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:33:42.090 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:33:42.090 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=0 2020-03-11T22:33:42.090 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:33:42.090 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/0 2020-03-11T22:33:42.090 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=cb7f2e42-1934-4145-a3c3-29bdd93f8adb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:33:42.090 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:33:42.090 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:33:42.091 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:33:42.091 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-11T22:33:42.091 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-11T22:33:42.091 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:33:42.091 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:33:42.091 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:33:42.091 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:33:42.092 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:33:42.092 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:33:42.092 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:33:42.092 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:33:42.092 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:33:42.092 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17931/$cluster-$name.asok' 2020-03-11T22:33:42.093 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' 2020-03-11T22:33:42.093 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:33:42.093 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:33:42.093 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:33:42.093 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:33:42.093 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:33:42.093 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:33:42.094 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:33:42.094 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:33:42.094 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/0 2020-03-11T22:33:42.094 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.0 2020-03-11T22:33:42.094 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 0 --fsid=cb7f2e42-1934-4145-a3c3-29bdd93f8adb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:33:42.094 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/0/whoami 2020-03-11T22:33:42.094 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.0 2020-03-11T22:33:42.095 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 0 = 0 ']' 2020-03-11T22:33:42.095 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:33:42.096 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:33:42.096 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:33:42.116 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:33:42.112+0000 7f4acf9ffec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:33:42.118 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:33:42.114+0000 7f4acf9ffec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:33:42.119 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:33:42.116+0000 7f4acf9ffec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:33:42.455 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 0 2020-03-11T22:33:42.456 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:33:42.456 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:33:42.456 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:33:42.456 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:33:42.456 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:33:42.457 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:33:42.457 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:33:42.457 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:33:42.457 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:33:42.635 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:33:42.631+0000 7f4acf9ffec0 -1 Falling back to public interface 2020-03-11T22:33:42.844 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:33:43.517 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:33:43.513+0000 7f4acf9ffec0 -1 osd.0 29 log_to_monitors {default=true} 2020-03-11T22:33:43.845 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:33:43.846 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:33:43.846 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:33:43.846 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:33:43.846 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:33:43.846 INFO:tasks.workunit.client.0.smithi059.stdout:1 2020-03-11T22:33:44.227 INFO:tasks.workunit.client.0.smithi059.stdout:osd.0 up out weight 0 up_from 35 up_thru 0 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/23083,v1:127.0.0.1:6803/23083] [v2:127.0.0.1:6804/23083,v1:127.0.0.1:6805/23083] exists,up 3024509c-3f4d-4606-be45-2b0ba42736ad 2020-03-11T22:33:44.228 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.2 2020-03-11T22:33:44.228 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:33:44.228 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:33:44.228 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:33:44.228 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:141: TEST_divergent: for i in $non_divergent 2020-03-11T22:33:44.228 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:143: TEST_divergent: activate_osd td/divergent-priors 2 2020-03-11T22:33:44.229 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:33:44.229 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:33:44.229 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=2 2020-03-11T22:33:44.229 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:33:44.229 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/2 2020-03-11T22:33:44.229 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=cb7f2e42-1934-4145-a3c3-29bdd93f8adb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:33:44.230 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:33:44.230 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:33:44.230 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:33:44.230 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-11T22:33:44.230 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-11T22:33:44.231 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:33:44.231 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:33:44.231 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:33:44.231 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:33:44.231 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:33:44.231 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:33:44.232 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:33:44.232 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:33:44.232 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:33:44.232 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17931/$cluster-$name.asok' 2020-03-11T22:33:44.232 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' 2020-03-11T22:33:44.232 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:33:44.232 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:33:44.233 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:33:44.233 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:33:44.233 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:33:44.233 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:33:44.233 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:33:44.233 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:33:44.233 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/2 2020-03-11T22:33:44.236 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.2 2020-03-11T22:33:44.237 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 2 --fsid=cb7f2e42-1934-4145-a3c3-29bdd93f8adb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:33:44.237 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/2/whoami 2020-03-11T22:33:44.238 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 2 = 2 ']' 2020-03-11T22:33:44.238 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:33:44.238 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:33:44.238 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:33:44.255 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:33:44.251+0000 7f60ea0c6ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:33:44.256 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:33:44.253+0000 7f60ea0c6ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:33:44.259 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:33:44.255+0000 7f60ea0c6ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:33:44.614 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 2 2020-03-11T22:33:44.615 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:33:44.615 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:33:44.615 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:33:44.615 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:33:44.615 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:33:44.616 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:33:44.616 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:33:44.616 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:33:44.616 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:33:44.776 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:33:44.772+0000 7f60ea0c6ec0 -1 Falling back to public interface 2020-03-11T22:33:44.980 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:33:45.625 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:33:45.621+0000 7f60ea0c6ec0 -1 osd.2 29 log_to_monitors {default=true} 2020-03-11T22:33:45.982 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:33:45.982 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:33:45.982 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:33:45.982 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:33:45.983 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:33:45.983 INFO:tasks.workunit.client.0.smithi059.stdout:1 2020-03-11T22:33:46.358 INFO:tasks.workunit.client.0.smithi059.stdout:osd.2 up out weight 0 up_from 36 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/23277,v1:127.0.0.1:6811/23277] [v2:127.0.0.1:6812/23277,v1:127.0.0.1:6813/23277] exists,up 9a5ddff4-d2b7-43cd-b7c4-5b04744a990c 2020-03-11T22:33:46.358 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:33:46.358 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:33:46.358 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:33:46.358 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:145: TEST_divergent: for i in $non_divergent 2020-03-11T22:33:46.359 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:147: TEST_divergent: ceph osd in osd.0 2020-03-11T22:33:46.892 INFO:tasks.workunit.client.0.smithi059.stderr:osd.0 is already in. 2020-03-11T22:33:46.900 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:145: TEST_divergent: for i in $non_divergent 2020-03-11T22:33:46.900 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:147: TEST_divergent: ceph osd in osd.2 2020-03-11T22:33:47.553 INFO:tasks.workunit.client.0.smithi059.stderr:osd.2 is already in. 2020-03-11T22:33:47.562 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:150: TEST_divergent: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:33:47.562 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:150: TEST_divergent: wait_for_clean 2020-03-11T22:33:47.562 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:33:47.562 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:33:47.562 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:33:47.563 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:33:47.563 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:33:47.564 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:33:47.564 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:33:47.564 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:33:47.564 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:33:47.565 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:33:47.629 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:33:47.629 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:33:47.629 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:33:47.629 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:33:47.629 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:33:47.987 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:33:47.987 INFO:tasks.workunit.client.0.smithi059.stderr:1 2020-03-11T22:33:47.987 INFO:tasks.workunit.client.0.smithi059.stderr:2' 2020-03-11T22:33:47.987 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:33:47.988 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:33:47.988 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:33:48.145 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=150323855363 2020-03-11T22:33:48.146 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-150323855363' 2020-03-11T22:33:48.146 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:33:48.146 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:33:48.241 INFO:tasks.workunit.client.0.smithi059.stderr:Error ENXIO: problem getting command descriptions from osd.1 2020-03-11T22:33:48.249 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq= 2020-03-11T22:33:48.250 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-150323855363 1-' 2020-03-11T22:33:48.250 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:33:48.250 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:33:48.385 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=154618822659 2020-03-11T22:33:48.385 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-150323855363 1- 2-154618822659' 2020-03-11T22:33:48.385 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:33:48.386 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-150323855363 2020-03-11T22:33:48.386 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:33:48.387 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:33:48.387 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-150323855363 2020-03-11T22:33:48.387 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:33:48.388 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=150323855363 2020-03-11T22:33:48.388 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 150323855363' 2020-03-11T22:33:48.388 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:33:48.388 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.0 seq 150323855363 2020-03-11T22:33:48.750 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 0 -lt 150323855363 2020-03-11T22:33:48.750 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:33:49.751 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:33:49.751 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:33:50.130 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 150323855364 -lt 150323855363 2020-03-11T22:33:50.130 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:33:50.131 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1- 2020-03-11T22:33:50.131 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:33:50.131 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:33:50.132 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1- 2020-03-11T22:33:50.132 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:33:50.132 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq= 2020-03-11T22:33:50.132 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq ' 2020-03-11T22:33:50.132 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:33:50.133 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.1 seq 2020-03-11T22:33:50.507 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672975 -lt 2020-03-11T22:33:50.508 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 2079: test: 42949672975: unary operator expected 2020-03-11T22:33:50.508 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:33:50.508 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-154618822659 2020-03-11T22:33:50.508 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:33:50.509 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:33:50.509 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-154618822659 2020-03-11T22:33:50.509 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:33:50.510 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=154618822659 2020-03-11T22:33:50.510 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 154618822659' 2020-03-11T22:33:50.510 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:33:50.511 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.2 seq 154618822659 2020-03-11T22:33:50.879 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 154618822659 -lt 154618822659 2020-03-11T22:33:50.879 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:33:50.880 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:33:50.880 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:33:51.390 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:33:51.390 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:33:51.390 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:33:51.391 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:33:51.391 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:33:51.391 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:33:51.391 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:33:51.391 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:33:51.742 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:33:51.742 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:33:51.743 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:33:51.743 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:33:52.215 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:33:52.215 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:33:52.215 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:33:52.216 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:33:52.216 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:33:52.216 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:33:52.315 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:33:52.315 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:33:52.316 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:33:52.316 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:33:52.317 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:33:52.317 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:33:52.317 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:33:52.317 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:33:52.667 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:33:52.667 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:33:52.668 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:33:52.668 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:33:53.139 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:33:53.140 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:33:53.140 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:33:53.140 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:33:53.140 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:33:53.140 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:33:53.140 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:33:53.140 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:33:53.141 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:33:53.611 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:33:53.611 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:33:53.611 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 1 >= 8 )) 2020-03-11T22:33:53.612 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:33:53.612 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.2 2020-03-11T22:33:53.812 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:33:53.813 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:33:53.813 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:33:53.813 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:33:53.813 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:33:53.815 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:33:53.815 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:33:53.815 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:33:54.164 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:33:54.165 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:33:54.165 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:33:54.165 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:33:54.641 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:33:54.641 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:33:54.642 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:33:54.642 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:33:54.642 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:33:54.642 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:33:54.642 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:33:54.642 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:33:54.643 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:33:55.114 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:33:55.114 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:33:55.114 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 2 >= 8 )) 2020-03-11T22:33:55.114 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:33:55.114 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.4 2020-03-11T22:33:55.515 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:33:55.515 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:33:55.515 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:33:55.516 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:33:55.516 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:33:55.516 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:33:55.516 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:33:55.516 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:33:55.870 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:33:55.870 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:33:55.871 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:33:55.871 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:33:56.340 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:33:56.340 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:33:56.340 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:33:56.340 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:33:56.341 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:33:56.341 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:33:56.341 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:33:56.341 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:33:56.341 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:33:56.812 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:33:56.812 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:33:56.812 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 3 >= 8 )) 2020-03-11T22:33:56.812 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:33:56.812 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.8 2020-03-11T22:33:57.613 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:33:57.613 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:33:57.615 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:33:57.617 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:33:57.617 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:33:57.617 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:33:57.617 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:33:57.617 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:33:57.979 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:33:57.979 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:33:57.980 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:33:57.980 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:33:58.451 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:33:58.451 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:33:58.451 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:33:58.452 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:33:58.452 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:33:58.452 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:33:58.452 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:33:58.453 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:33:58.453 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:33:58.918 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:33:58.918 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:33:58.918 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 4 >= 8 )) 2020-03-11T22:33:58.918 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:33:58.918 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 1.6 2020-03-11T22:34:00.519 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:34:00.519 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:34:00.520 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:34:00.520 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:34:00.520 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:34:00.520 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:34:00.520 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:34:00.520 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:34:00.882 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:34:00.882 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:34:00.883 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:34:00.883 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:34:01.352 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:34:01.352 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:34:01.352 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:34:01.352 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:34:01.352 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:34:01.352 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:34:01.353 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:34:01.353 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:34:01.353 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:34:01.818 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:34:01.819 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:34:01.819 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 5 >= 8 )) 2020-03-11T22:34:01.819 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:34:01.819 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 3.2 2020-03-11T22:34:05.019 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:34:05.020 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:34:05.020 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:34:05.021 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:34:05.021 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:34:05.021 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:34:05.021 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:34:05.021 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:34:05.383 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:34:05.383 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:34:05.384 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:34:05.384 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:34:05.873 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:34:05.873 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:34:05.873 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:34:05.873 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:34:05.873 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:34:05.873 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:34:05.874 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:34:05.874 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:34:05.874 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:34:06.344 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:34:06.344 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:34:06.344 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 6 >= 8 )) 2020-03-11T22:34:06.344 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:34:06.344 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 6.4 2020-03-11T22:34:12.744 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:34:12.745 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:34:12.746 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:34:12.746 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:34:12.746 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:34:12.746 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:34:12.746 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:34:12.746 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:34:13.111 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:34:13.112 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:34:13.112 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:34:13.112 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:34:13.587 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:34:13.587 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:34:13.587 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:34:13.587 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:34:13.587 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:34:13.587 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:34:13.588 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:34:13.588 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:34:13.588 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:34:14.053 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:34:14.053 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:34:14.053 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 7 >= 8 )) 2020-03-11T22:34:14.054 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:34:14.054 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 7.3 2020-03-11T22:34:21.354 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:34:21.354 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:34:21.355 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:34:21.355 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:34:21.355 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:34:21.355 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:34:21.355 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:34:21.356 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:34:21.721 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:34:21.721 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:34:21.722 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:34:21.722 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:34:22.190 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:34:22.191 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:34:22.191 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:34:22.191 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:34:22.191 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:34:22.191 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:34:22.191 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:34:22.191 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:34:22.192 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:34:22.669 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:34:22.669 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:34:22.669 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 8 >= 8 )) 2020-03-11T22:34:22.669 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: ceph report 2020-03-11T22:34:23.131 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2020-03-11T22:34:23.132 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_fingerprint": "1dccad07-27e8-48ea-a9ba-514ff7bcf351", 2020-03-11T22:34:23.132 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "15.1.0-1941-g4acc112", 2020-03-11T22:34:23.132 INFO:tasks.workunit.client.0.smithi059.stdout: "commit": "4acc1121e47039a3cdf2978ae458c10d153aa141", 2020-03-11T22:34:23.132 INFO:tasks.workunit.client.0.smithi059.stdout: "timestamp": "2020-03-11T22:34:23.128236+0000", 2020-03-11T22:34:23.132 INFO:tasks.workunit.client.0.smithi059.stdout: "tag": "", 2020-03-11T22:34:23.133 INFO:tasks.workunit.client.0.smithi059.stdout: "health": { 2020-03-11T22:34:23.133 INFO:tasks.workunit.client.0.smithi059.stdout: "status": "HEALTH_WARN", 2020-03-11T22:34:23.133 INFO:tasks.workunit.client.0.smithi059.stdout: "checks": { 2020-03-11T22:34:23.133 INFO:tasks.workunit.client.0.smithi059.stdout: "OSDMAP_FLAGS": { 2020-03-11T22:34:23.133 INFO:tasks.workunit.client.0.smithi059.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:34:23.133 INFO:tasks.workunit.client.0.smithi059.stdout: "summary": { 2020-03-11T22:34:23.133 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "nodown,noout,noin flag(s) set", 2020-03-11T22:34:23.134 INFO:tasks.workunit.client.0.smithi059.stdout: "count": 17 2020-03-11T22:34:23.134 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.134 INFO:tasks.workunit.client.0.smithi059.stdout: "detail": [], 2020-03-11T22:34:23.134 INFO:tasks.workunit.client.0.smithi059.stdout: "muted": false 2020-03-11T22:34:23.134 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.134 INFO:tasks.workunit.client.0.smithi059.stdout: "OSD_DOWN": { 2020-03-11T22:34:23.135 INFO:tasks.workunit.client.0.smithi059.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:34:23.135 INFO:tasks.workunit.client.0.smithi059.stdout: "summary": { 2020-03-11T22:34:23.135 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "1 osds down", 2020-03-11T22:34:23.135 INFO:tasks.workunit.client.0.smithi059.stdout: "count": 1 2020-03-11T22:34:23.135 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.135 INFO:tasks.workunit.client.0.smithi059.stdout: "detail": [ 2020-03-11T22:34:23.135 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.136 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "osd.1 (root=default,host=smithi059) is down" 2020-03-11T22:34:23.136 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:34:23.136 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:34:23.136 INFO:tasks.workunit.client.0.smithi059.stdout: "muted": false 2020-03-11T22:34:23.136 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.136 INFO:tasks.workunit.client.0.smithi059.stdout: "PG_DEGRADED": { 2020-03-11T22:34:23.136 INFO:tasks.workunit.client.0.smithi059.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:34:23.137 INFO:tasks.workunit.client.0.smithi059.stdout: "summary": { 2020-03-11T22:34:23.137 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded", 2020-03-11T22:34:23.137 INFO:tasks.workunit.client.0.smithi059.stdout: "count": 1 2020-03-11T22:34:23.137 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.137 INFO:tasks.workunit.client.0.smithi059.stdout: "detail": [ 2020-03-11T22:34:23.137 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.138 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:34:23.138 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:34:23.138 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:34:23.138 INFO:tasks.workunit.client.0.smithi059.stdout: "muted": false 2020-03-11T22:34:23.138 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.138 INFO:tasks.workunit.client.0.smithi059.stdout: "POOL_APP_NOT_ENABLED": { 2020-03-11T22:34:23.138 INFO:tasks.workunit.client.0.smithi059.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:34:23.139 INFO:tasks.workunit.client.0.smithi059.stdout: "summary": { 2020-03-11T22:34:23.139 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "1 pool(s) do not have an application enabled", 2020-03-11T22:34:23.139 INFO:tasks.workunit.client.0.smithi059.stdout: "count": 1 2020-03-11T22:34:23.139 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.139 INFO:tasks.workunit.client.0.smithi059.stdout: "detail": [ 2020-03-11T22:34:23.139 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.139 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "application not enabled on pool 'test'" 2020-03-11T22:34:23.140 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.140 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.140 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2020-03-11T22:34:23.140 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:34:23.140 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:34:23.140 INFO:tasks.workunit.client.0.smithi059.stdout: "muted": false 2020-03-11T22:34:23.141 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:34:23.141 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.141 INFO:tasks.workunit.client.0.smithi059.stdout: "mutes": [] 2020-03-11T22:34:23.141 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.141 INFO:tasks.workunit.client.0.smithi059.stdout: "monmap_first_committed": 1, 2020-03-11T22:34:23.141 INFO:tasks.workunit.client.0.smithi059.stdout: "monmap_last_committed": 1, 2020-03-11T22:34:23.141 INFO:tasks.workunit.client.0.smithi059.stdout: "monmap": { 2020-03-11T22:34:23.142 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": 1, 2020-03-11T22:34:23.142 INFO:tasks.workunit.client.0.smithi059.stdout: "fsid": "cb7f2e42-1934-4145-a3c3-29bdd93f8adb", 2020-03-11T22:34:23.142 INFO:tasks.workunit.client.0.smithi059.stdout: "modified": "2020-03-11T22:32:41.475493Z", 2020-03-11T22:34:23.142 INFO:tasks.workunit.client.0.smithi059.stdout: "created": "2020-03-11T22:32:41.475493Z", 2020-03-11T22:34:23.142 INFO:tasks.workunit.client.0.smithi059.stdout: "min_mon_release": 15, 2020-03-11T22:34:23.142 INFO:tasks.workunit.client.0.smithi059.stdout: "min_mon_release_name": "octopus", 2020-03-11T22:34:23.142 INFO:tasks.workunit.client.0.smithi059.stdout: "features": { 2020-03-11T22:34:23.143 INFO:tasks.workunit.client.0.smithi059.stdout: "persistent": [ 2020-03-11T22:34:23.143 INFO:tasks.workunit.client.0.smithi059.stdout: "kraken", 2020-03-11T22:34:23.143 INFO:tasks.workunit.client.0.smithi059.stdout: "luminous", 2020-03-11T22:34:23.143 INFO:tasks.workunit.client.0.smithi059.stdout: "mimic", 2020-03-11T22:34:23.143 INFO:tasks.workunit.client.0.smithi059.stdout: "osdmap-prune", 2020-03-11T22:34:23.143 INFO:tasks.workunit.client.0.smithi059.stdout: "nautilus", 2020-03-11T22:34:23.144 INFO:tasks.workunit.client.0.smithi059.stdout: "octopus" 2020-03-11T22:34:23.144 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:34:23.144 INFO:tasks.workunit.client.0.smithi059.stdout: "optional": [] 2020-03-11T22:34:23.144 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.144 INFO:tasks.workunit.client.0.smithi059.stdout: "mons": [ 2020-03-11T22:34:23.144 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.144 INFO:tasks.workunit.client.0.smithi059.stdout: "rank": 0, 2020-03-11T22:34:23.145 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "a", 2020-03-11T22:34:23.145 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addrs": { 2020-03-11T22:34:23.145 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:34:23.145 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.145 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:34:23.145 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:7115", 2020-03-11T22:34:23.145 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 0 2020-03-11T22:34:23.146 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:34:23.146 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:34:23.146 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.146 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:7115/0", 2020-03-11T22:34:23.146 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addr": "127.0.0.1:7115/0", 2020-03-11T22:34:23.146 INFO:tasks.workunit.client.0.smithi059.stdout: "priority": 0, 2020-03-11T22:34:23.146 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 0 2020-03-11T22:34:23.147 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:34:23.147 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:34:23.147 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.147 INFO:tasks.workunit.client.0.smithi059.stdout: "quorum": [ 2020-03-11T22:34:23.147 INFO:tasks.workunit.client.0.smithi059.stdout: 0 2020-03-11T22:34:23.147 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:34:23.148 INFO:tasks.workunit.client.0.smithi059.stdout: "osdmap": { 2020-03-11T22:34:23.148 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": 39, 2020-03-11T22:34:23.148 INFO:tasks.workunit.client.0.smithi059.stdout: "fsid": "cb7f2e42-1934-4145-a3c3-29bdd93f8adb", 2020-03-11T22:34:23.148 INFO:tasks.workunit.client.0.smithi059.stdout: "created": "2020-03-11T22:32:42.368343+0000", 2020-03-11T22:34:23.148 INFO:tasks.workunit.client.0.smithi059.stdout: "modified": "2020-03-11T22:33:47.455890+0000", 2020-03-11T22:34:23.148 INFO:tasks.workunit.client.0.smithi059.stdout: "last_up_change": "2020-03-11T22:33:45.703337+0000", 2020-03-11T22:34:23.148 INFO:tasks.workunit.client.0.smithi059.stdout: "last_in_change": "2020-03-11T22:33:47.255454+0000", 2020-03-11T22:34:23.149 INFO:tasks.workunit.client.0.smithi059.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-03-11T22:34:23.149 INFO:tasks.workunit.client.0.smithi059.stdout: "flags_num": 5800384, 2020-03-11T22:34:23.149 INFO:tasks.workunit.client.0.smithi059.stdout: "flags_set": [ 2020-03-11T22:34:23.149 INFO:tasks.workunit.client.0.smithi059.stdout: "nodown", 2020-03-11T22:34:23.149 INFO:tasks.workunit.client.0.smithi059.stdout: "noin", 2020-03-11T22:34:23.149 INFO:tasks.workunit.client.0.smithi059.stdout: "noout", 2020-03-11T22:34:23.149 INFO:tasks.workunit.client.0.smithi059.stdout: "pglog_hardlimit", 2020-03-11T22:34:23.150 INFO:tasks.workunit.client.0.smithi059.stdout: "purged_snapdirs", 2020-03-11T22:34:23.150 INFO:tasks.workunit.client.0.smithi059.stdout: "recovery_deletes", 2020-03-11T22:34:23.150 INFO:tasks.workunit.client.0.smithi059.stdout: "sortbitwise" 2020-03-11T22:34:23.150 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:34:23.150 INFO:tasks.workunit.client.0.smithi059.stdout: "crush_version": 7, 2020-03-11T22:34:23.150 INFO:tasks.workunit.client.0.smithi059.stdout: "full_ratio": 0.99000000953674316, 2020-03-11T22:34:23.151 INFO:tasks.workunit.client.0.smithi059.stdout: "backfillfull_ratio": 0.99000000953674316, 2020-03-11T22:34:23.151 INFO:tasks.workunit.client.0.smithi059.stdout: "nearfull_ratio": 0.85000002384185791, 2020-03-11T22:34:23.151 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_snapshot": "", 2020-03-11T22:34:23.151 INFO:tasks.workunit.client.0.smithi059.stdout: "pool_max": 1, 2020-03-11T22:34:23.151 INFO:tasks.workunit.client.0.smithi059.stdout: "max_osd": 3, 2020-03-11T22:34:23.151 INFO:tasks.workunit.client.0.smithi059.stdout: "require_min_compat_client": "jewel", 2020-03-11T22:34:23.151 INFO:tasks.workunit.client.0.smithi059.stdout: "min_compat_client": "jewel", 2020-03-11T22:34:23.152 INFO:tasks.workunit.client.0.smithi059.stdout: "require_osd_release": "octopus", 2020-03-11T22:34:23.152 INFO:tasks.workunit.client.0.smithi059.stdout: "pools": [ 2020-03-11T22:34:23.152 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.152 INFO:tasks.workunit.client.0.smithi059.stdout: "pool": 1, 2020-03-11T22:34:23.152 INFO:tasks.workunit.client.0.smithi059.stdout: "pool_name": "test", 2020-03-11T22:34:23.152 INFO:tasks.workunit.client.0.smithi059.stdout: "create_time": "2020-03-11T22:33:04.314682+0000", 2020-03-11T22:34:23.152 INFO:tasks.workunit.client.0.smithi059.stdout: "flags": 1, 2020-03-11T22:34:23.153 INFO:tasks.workunit.client.0.smithi059.stdout: "flags_names": "hashpspool", 2020-03-11T22:34:23.153 INFO:tasks.workunit.client.0.smithi059.stdout: "type": 1, 2020-03-11T22:34:23.153 INFO:tasks.workunit.client.0.smithi059.stdout: "size": 3, 2020-03-11T22:34:23.153 INFO:tasks.workunit.client.0.smithi059.stdout: "min_size": 2, 2020-03-11T22:34:23.153 INFO:tasks.workunit.client.0.smithi059.stdout: "crush_rule": 0, 2020-03-11T22:34:23.153 INFO:tasks.workunit.client.0.smithi059.stdout: "object_hash": 2, 2020-03-11T22:34:23.153 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_autoscale_mode": "off", 2020-03-11T22:34:23.154 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_num": 1, 2020-03-11T22:34:23.154 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_placement_num": 1, 2020-03-11T22:34:23.154 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_placement_num_target": 1, 2020-03-11T22:34:23.154 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_num_target": 1, 2020-03-11T22:34:23.154 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_num_pending": 1, 2020-03-11T22:34:23.154 INFO:tasks.workunit.client.0.smithi059.stdout: "last_pg_merge_meta": { 2020-03-11T22:34:23.155 INFO:tasks.workunit.client.0.smithi059.stdout: "source_pgid": "0.0", 2020-03-11T22:34:23.155 INFO:tasks.workunit.client.0.smithi059.stdout: "ready_epoch": 0, 2020-03-11T22:34:23.155 INFO:tasks.workunit.client.0.smithi059.stdout: "last_epoch_started": 0, 2020-03-11T22:34:23.155 INFO:tasks.workunit.client.0.smithi059.stdout: "last_epoch_clean": 0, 2020-03-11T22:34:23.155 INFO:tasks.workunit.client.0.smithi059.stdout: "source_version": "0'0", 2020-03-11T22:34:23.155 INFO:tasks.workunit.client.0.smithi059.stdout: "target_version": "0'0" 2020-03-11T22:34:23.155 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.156 INFO:tasks.workunit.client.0.smithi059.stdout: "last_change": "29", 2020-03-11T22:34:23.156 INFO:tasks.workunit.client.0.smithi059.stdout: "last_force_op_resend": "0", 2020-03-11T22:34:23.156 INFO:tasks.workunit.client.0.smithi059.stdout: "last_force_op_resend_prenautilus": "0", 2020-03-11T22:34:23.156 INFO:tasks.workunit.client.0.smithi059.stdout: "last_force_op_resend_preluminous": "0", 2020-03-11T22:34:23.156 INFO:tasks.workunit.client.0.smithi059.stdout: "auid": 0, 2020-03-11T22:34:23.156 INFO:tasks.workunit.client.0.smithi059.stdout: "snap_mode": "selfmanaged", 2020-03-11T22:34:23.156 INFO:tasks.workunit.client.0.smithi059.stdout: "snap_seq": 0, 2020-03-11T22:34:23.157 INFO:tasks.workunit.client.0.smithi059.stdout: "snap_epoch": 0, 2020-03-11T22:34:23.157 INFO:tasks.workunit.client.0.smithi059.stdout: "pool_snaps": [], 2020-03-11T22:34:23.157 INFO:tasks.workunit.client.0.smithi059.stdout: "removed_snaps": "[]", 2020-03-11T22:34:23.157 INFO:tasks.workunit.client.0.smithi059.stdout: "quota_max_bytes": 0, 2020-03-11T22:34:23.157 INFO:tasks.workunit.client.0.smithi059.stdout: "quota_max_objects": 0, 2020-03-11T22:34:23.157 INFO:tasks.workunit.client.0.smithi059.stdout: "tiers": [], 2020-03-11T22:34:23.158 INFO:tasks.workunit.client.0.smithi059.stdout: "tier_of": -1, 2020-03-11T22:34:23.158 INFO:tasks.workunit.client.0.smithi059.stdout: "read_tier": -1, 2020-03-11T22:34:23.158 INFO:tasks.workunit.client.0.smithi059.stdout: "write_tier": -1, 2020-03-11T22:34:23.158 INFO:tasks.workunit.client.0.smithi059.stdout: "cache_mode": "none", 2020-03-11T22:34:23.158 INFO:tasks.workunit.client.0.smithi059.stdout: "target_max_bytes": 0, 2020-03-11T22:34:23.158 INFO:tasks.workunit.client.0.smithi059.stdout: "target_max_objects": 0, 2020-03-11T22:34:23.158 INFO:tasks.workunit.client.0.smithi059.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-03-11T22:34:23.159 INFO:tasks.workunit.client.0.smithi059.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-03-11T22:34:23.159 INFO:tasks.workunit.client.0.smithi059.stdout: "cache_target_full_ratio_micro": 800000, 2020-03-11T22:34:23.159 INFO:tasks.workunit.client.0.smithi059.stdout: "cache_min_flush_age": 0, 2020-03-11T22:34:23.159 INFO:tasks.workunit.client.0.smithi059.stdout: "cache_min_evict_age": 0, 2020-03-11T22:34:23.159 INFO:tasks.workunit.client.0.smithi059.stdout: "erasure_code_profile": "", 2020-03-11T22:34:23.159 INFO:tasks.workunit.client.0.smithi059.stdout: "hit_set_params": { 2020-03-11T22:34:23.159 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "none" 2020-03-11T22:34:23.160 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.160 INFO:tasks.workunit.client.0.smithi059.stdout: "hit_set_period": 0, 2020-03-11T22:34:23.160 INFO:tasks.workunit.client.0.smithi059.stdout: "hit_set_count": 0, 2020-03-11T22:34:23.160 INFO:tasks.workunit.client.0.smithi059.stdout: "use_gmt_hitset": true, 2020-03-11T22:34:23.160 INFO:tasks.workunit.client.0.smithi059.stdout: "min_read_recency_for_promote": 0, 2020-03-11T22:34:23.160 INFO:tasks.workunit.client.0.smithi059.stdout: "min_write_recency_for_promote": 0, 2020-03-11T22:34:23.160 INFO:tasks.workunit.client.0.smithi059.stdout: "hit_set_grade_decay_rate": 0, 2020-03-11T22:34:23.161 INFO:tasks.workunit.client.0.smithi059.stdout: "hit_set_search_last_n": 0, 2020-03-11T22:34:23.161 INFO:tasks.workunit.client.0.smithi059.stdout: "grade_table": [], 2020-03-11T22:34:23.161 INFO:tasks.workunit.client.0.smithi059.stdout: "stripe_width": 0, 2020-03-11T22:34:23.161 INFO:tasks.workunit.client.0.smithi059.stdout: "expected_num_objects": 0, 2020-03-11T22:34:23.161 INFO:tasks.workunit.client.0.smithi059.stdout: "fast_read": false, 2020-03-11T22:34:23.161 INFO:tasks.workunit.client.0.smithi059.stdout: "options": {}, 2020-03-11T22:34:23.162 INFO:tasks.workunit.client.0.smithi059.stdout: "application_metadata": {} 2020-03-11T22:34:23.162 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:34:23.162 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:34:23.162 INFO:tasks.workunit.client.0.smithi059.stdout: "osds": [ 2020-03-11T22:34:23.162 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.162 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 0, 2020-03-11T22:34:23.162 INFO:tasks.workunit.client.0.smithi059.stdout: "uuid": "3024509c-3f4d-4606-be45-2b0ba42736ad", 2020-03-11T22:34:23.163 INFO:tasks.workunit.client.0.smithi059.stdout: "up": 1, 2020-03-11T22:34:23.163 INFO:tasks.workunit.client.0.smithi059.stdout: "in": 1, 2020-03-11T22:34:23.163 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 1, 2020-03-11T22:34:23.163 INFO:tasks.workunit.client.0.smithi059.stdout: "primary_affinity": 1, 2020-03-11T22:34:23.163 INFO:tasks.workunit.client.0.smithi059.stdout: "last_clean_begin": 5, 2020-03-11T22:34:23.163 INFO:tasks.workunit.client.0.smithi059.stdout: "last_clean_end": 29, 2020-03-11T22:34:23.163 INFO:tasks.workunit.client.0.smithi059.stdout: "up_from": 35, 2020-03-11T22:34:23.164 INFO:tasks.workunit.client.0.smithi059.stdout: "up_thru": 38, 2020-03-11T22:34:23.164 INFO:tasks.workunit.client.0.smithi059.stdout: "down_at": 30, 2020-03-11T22:34:23.164 INFO:tasks.workunit.client.0.smithi059.stdout: "lost_at": 0, 2020-03-11T22:34:23.164 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addrs": { 2020-03-11T22:34:23.164 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:34:23.164 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.164 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:34:23.165 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6802", 2020-03-11T22:34:23.165 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 23083 2020-03-11T22:34:23.165 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.165 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.165 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:34:23.166 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6803", 2020-03-11T22:34:23.166 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 23083 2020-03-11T22:34:23.166 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:34:23.166 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:34:23.166 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.166 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_addrs": { 2020-03-11T22:34:23.166 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:34:23.167 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.167 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:34:23.167 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6804", 2020-03-11T22:34:23.167 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 23083 2020-03-11T22:34:23.167 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.167 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.167 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:34:23.168 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6805", 2020-03-11T22:34:23.168 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 23083 2020-03-11T22:34:23.168 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:34:23.168 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:34:23.168 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.169 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_back_addrs": { 2020-03-11T22:34:23.169 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:34:23.169 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.169 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:34:23.169 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6808", 2020-03-11T22:34:23.169 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 23083 2020-03-11T22:34:23.169 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.170 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.170 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:34:23.170 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6809", 2020-03-11T22:34:23.170 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 23083 2020-03-11T22:34:23.170 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:34:23.170 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:34:23.170 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.171 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_front_addrs": { 2020-03-11T22:34:23.171 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:34:23.171 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.171 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:34:23.171 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6806", 2020-03-11T22:34:23.171 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 23083 2020-03-11T22:34:23.171 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.172 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.172 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:34:23.172 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6807", 2020-03-11T22:34:23.172 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 23083 2020-03-11T22:34:23.172 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:34:23.172 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:34:23.173 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.173 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addr": "127.0.0.1:6803/23083", 2020-03-11T22:34:23.173 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_addr": "127.0.0.1:6805/23083", 2020-03-11T22:34:23.173 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_back_addr": "127.0.0.1:6809/23083", 2020-03-11T22:34:23.173 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_front_addr": "127.0.0.1:6807/23083", 2020-03-11T22:34:23.173 INFO:tasks.workunit.client.0.smithi059.stdout: "state": [ 2020-03-11T22:34:23.173 INFO:tasks.workunit.client.0.smithi059.stdout: "exists", 2020-03-11T22:34:23.174 INFO:tasks.workunit.client.0.smithi059.stdout: "up" 2020-03-11T22:34:23.174 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:34:23.174 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.174 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.174 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 1, 2020-03-11T22:34:23.174 INFO:tasks.workunit.client.0.smithi059.stdout: "uuid": "85494619-d02f-47a9-a790-661f9ba109cb", 2020-03-11T22:34:23.174 INFO:tasks.workunit.client.0.smithi059.stdout: "up": 0, 2020-03-11T22:34:23.175 INFO:tasks.workunit.client.0.smithi059.stdout: "in": 1, 2020-03-11T22:34:23.175 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 1, 2020-03-11T22:34:23.175 INFO:tasks.workunit.client.0.smithi059.stdout: "primary_affinity": 1, 2020-03-11T22:34:23.175 INFO:tasks.workunit.client.0.smithi059.stdout: "last_clean_begin": 0, 2020-03-11T22:34:23.175 INFO:tasks.workunit.client.0.smithi059.stdout: "last_clean_end": 0, 2020-03-11T22:34:23.175 INFO:tasks.workunit.client.0.smithi059.stdout: "up_from": 10, 2020-03-11T22:34:23.175 INFO:tasks.workunit.client.0.smithi059.stdout: "up_thru": 23, 2020-03-11T22:34:23.176 INFO:tasks.workunit.client.0.smithi059.stdout: "down_at": 31, 2020-03-11T22:34:23.176 INFO:tasks.workunit.client.0.smithi059.stdout: "lost_at": 0, 2020-03-11T22:34:23.176 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addrs": { 2020-03-11T22:34:23.176 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:34:23.176 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.176 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:34:23.176 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:34:23.177 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 18723 2020-03-11T22:34:23.177 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.177 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.177 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:34:23.177 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:34:23.177 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 18723 2020-03-11T22:34:23.177 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:34:23.178 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:34:23.178 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.178 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_addrs": { 2020-03-11T22:34:23.178 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:34:23.178 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.178 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:34:23.179 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:34:23.179 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 18723 2020-03-11T22:34:23.179 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.179 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.179 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:34:23.179 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:34:23.179 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 18723 2020-03-11T22:34:23.180 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:34:23.180 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:34:23.180 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.180 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_back_addrs": { 2020-03-11T22:34:23.180 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:34:23.180 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.180 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:34:23.181 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:34:23.181 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 18723 2020-03-11T22:34:23.181 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.181 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.181 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:34:23.181 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:34:23.181 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 18723 2020-03-11T22:34:23.182 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:34:23.182 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:34:23.182 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.182 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_front_addrs": { 2020-03-11T22:34:23.182 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:34:23.182 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.182 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:34:23.183 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:34:23.183 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 18723 2020-03-11T22:34:23.183 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.183 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.183 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:34:23.183 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:34:23.183 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 18723 2020-03-11T22:34:23.184 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:34:23.184 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:34:23.184 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.184 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addr": "127.0.0.1:6811/18723", 2020-03-11T22:34:23.184 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_addr": "127.0.0.1:6813/18723", 2020-03-11T22:34:23.184 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_back_addr": "127.0.0.1:6817/18723", 2020-03-11T22:34:23.184 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_front_addr": "127.0.0.1:6815/18723", 2020-03-11T22:34:23.185 INFO:tasks.workunit.client.0.smithi059.stdout: "state": [ 2020-03-11T22:34:23.185 INFO:tasks.workunit.client.0.smithi059.stdout: "exists" 2020-03-11T22:34:23.185 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:34:23.185 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.185 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.185 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 2, 2020-03-11T22:34:23.185 INFO:tasks.workunit.client.0.smithi059.stdout: "uuid": "9a5ddff4-d2b7-43cd-b7c4-5b04744a990c", 2020-03-11T22:34:23.185 INFO:tasks.workunit.client.0.smithi059.stdout: "up": 1, 2020-03-11T22:34:23.186 INFO:tasks.workunit.client.0.smithi059.stdout: "in": 1, 2020-03-11T22:34:23.186 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 1, 2020-03-11T22:34:23.186 INFO:tasks.workunit.client.0.smithi059.stdout: "primary_affinity": 1, 2020-03-11T22:34:23.186 INFO:tasks.workunit.client.0.smithi059.stdout: "last_clean_begin": 15, 2020-03-11T22:34:23.186 INFO:tasks.workunit.client.0.smithi059.stdout: "last_clean_end": 29, 2020-03-11T22:34:23.186 INFO:tasks.workunit.client.0.smithi059.stdout: "up_from": 36, 2020-03-11T22:34:23.186 INFO:tasks.workunit.client.0.smithi059.stdout: "up_thru": 0, 2020-03-11T22:34:23.187 INFO:tasks.workunit.client.0.smithi059.stdout: "down_at": 32, 2020-03-11T22:34:23.187 INFO:tasks.workunit.client.0.smithi059.stdout: "lost_at": 0, 2020-03-11T22:34:23.187 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addrs": { 2020-03-11T22:34:23.187 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:34:23.187 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.187 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:34:23.187 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:34:23.187 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 23277 2020-03-11T22:34:23.188 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.188 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.188 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:34:23.188 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:34:23.188 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 23277 2020-03-11T22:34:23.188 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:34:23.188 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:34:23.189 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.189 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_addrs": { 2020-03-11T22:34:23.189 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:34:23.189 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.189 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:34:23.189 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:34:23.189 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 23277 2020-03-11T22:34:23.189 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.190 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.190 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:34:23.190 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:34:23.190 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 23277 2020-03-11T22:34:23.190 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:34:23.190 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:34:23.190 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.191 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_back_addrs": { 2020-03-11T22:34:23.191 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:34:23.191 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.191 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:34:23.191 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:34:23.191 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 23277 2020-03-11T22:34:23.191 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.192 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.192 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:34:23.192 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:34:23.192 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 23277 2020-03-11T22:34:23.192 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:34:23.192 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:34:23.192 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.193 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_front_addrs": { 2020-03-11T22:34:23.193 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:34:23.193 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.193 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:34:23.193 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:34:23.193 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 23277 2020-03-11T22:34:23.193 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.194 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.194 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:34:23.194 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:34:23.194 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 23277 2020-03-11T22:34:23.194 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:34:23.194 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:34:23.194 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.195 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addr": "127.0.0.1:6811/23277", 2020-03-11T22:34:23.195 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_addr": "127.0.0.1:6813/23277", 2020-03-11T22:34:23.195 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_back_addr": "127.0.0.1:6817/23277", 2020-03-11T22:34:23.195 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_front_addr": "127.0.0.1:6815/23277", 2020-03-11T22:34:23.195 INFO:tasks.workunit.client.0.smithi059.stdout: "state": [ 2020-03-11T22:34:23.195 INFO:tasks.workunit.client.0.smithi059.stdout: "exists", 2020-03-11T22:34:23.195 INFO:tasks.workunit.client.0.smithi059.stdout: "up" 2020-03-11T22:34:23.195 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:34:23.196 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:34:23.196 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:34:23.196 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_xinfo": [ 2020-03-11T22:34:23.196 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.196 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 0, 2020-03-11T22:34:23.196 INFO:tasks.workunit.client.0.smithi059.stdout: "down_stamp": "2020-03-11T22:33:39.354019+0000", 2020-03-11T22:34:23.196 INFO:tasks.workunit.client.0.smithi059.stdout: "laggy_probability": 0, 2020-03-11T22:34:23.197 INFO:tasks.workunit.client.0.smithi059.stdout: "laggy_interval": 0, 2020-03-11T22:34:23.197 INFO:tasks.workunit.client.0.smithi059.stdout: "features": 4540138292836696063, 2020-03-11T22:34:23.197 INFO:tasks.workunit.client.0.smithi059.stdout: "old_weight": 0, 2020-03-11T22:34:23.197 INFO:tasks.workunit.client.0.smithi059.stdout: "last_purged_snaps_scrub": "2020-03-11T22:32:49.191113+0000", 2020-03-11T22:34:23.197 INFO:tasks.workunit.client.0.smithi059.stdout: "dead_epoch": 0 2020-03-11T22:34:23.197 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.197 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.197 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 1, 2020-03-11T22:34:23.198 INFO:tasks.workunit.client.0.smithi059.stdout: "down_stamp": "2020-03-11T22:33:39.912427+0000", 2020-03-11T22:34:23.198 INFO:tasks.workunit.client.0.smithi059.stdout: "laggy_probability": 0, 2020-03-11T22:34:23.200 INFO:tasks.workunit.client.0.smithi059.stdout: "laggy_interval": 0, 2020-03-11T22:34:23.200 INFO:tasks.workunit.client.0.smithi059.stdout: "features": 4540138292836696063, 2020-03-11T22:34:23.201 INFO:tasks.workunit.client.0.smithi059.stdout: "old_weight": 0, 2020-03-11T22:34:23.201 INFO:tasks.workunit.client.0.smithi059.stdout: "last_purged_snaps_scrub": "2020-03-11T22:32:55.570478+0000", 2020-03-11T22:34:23.201 INFO:tasks.workunit.client.0.smithi059.stdout: "dead_epoch": 0 2020-03-11T22:34:23.201 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.201 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.201 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 2, 2020-03-11T22:34:23.202 INFO:tasks.workunit.client.0.smithi059.stdout: "down_stamp": "2020-03-11T22:33:40.464450+0000", 2020-03-11T22:34:23.202 INFO:tasks.workunit.client.0.smithi059.stdout: "laggy_probability": 0, 2020-03-11T22:34:23.202 INFO:tasks.workunit.client.0.smithi059.stdout: "laggy_interval": 0, 2020-03-11T22:34:23.202 INFO:tasks.workunit.client.0.smithi059.stdout: "features": 4540138292836696063, 2020-03-11T22:34:23.202 INFO:tasks.workunit.client.0.smithi059.stdout: "old_weight": 0, 2020-03-11T22:34:23.203 INFO:tasks.workunit.client.0.smithi059.stdout: "last_purged_snaps_scrub": "2020-03-11T22:33:01.936385+0000", 2020-03-11T22:34:23.203 INFO:tasks.workunit.client.0.smithi059.stdout: "dead_epoch": 0 2020-03-11T22:34:23.203 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:34:23.203 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:34:23.203 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_upmap": [], 2020-03-11T22:34:23.203 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_upmap_items": [], 2020-03-11T22:34:23.203 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_temp": [], 2020-03-11T22:34:23.204 INFO:tasks.workunit.client.0.smithi059.stdout: "primary_temp": [], 2020-03-11T22:34:23.204 INFO:tasks.workunit.client.0.smithi059.stdout: "blacklist": {}, 2020-03-11T22:34:23.204 INFO:tasks.workunit.client.0.smithi059.stdout: "erasure_code_profiles": { 2020-03-11T22:34:23.204 INFO:tasks.workunit.client.0.smithi059.stdout: "default": { 2020-03-11T22:34:23.204 INFO:tasks.workunit.client.0.smithi059.stdout: "k": "2", 2020-03-11T22:34:23.204 INFO:tasks.workunit.client.0.smithi059.stdout: "m": "2", 2020-03-11T22:34:23.204 INFO:tasks.workunit.client.0.smithi059.stdout: "plugin": "jerasure", 2020-03-11T22:34:23.205 INFO:tasks.workunit.client.0.smithi059.stdout: "technique": "reed_sol_van" 2020-03-11T22:34:23.205 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:34:23.205 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.205 INFO:tasks.workunit.client.0.smithi059.stdout: "removed_snaps_queue": [], 2020-03-11T22:34:23.205 INFO:tasks.workunit.client.0.smithi059.stdout: "new_removed_snaps": [], 2020-03-11T22:34:23.205 INFO:tasks.workunit.client.0.smithi059.stdout: "new_purged_snaps": [], 2020-03-11T22:34:23.205 INFO:tasks.workunit.client.0.smithi059.stdout: "crush_node_flags": {}, 2020-03-11T22:34:23.206 INFO:tasks.workunit.client.0.smithi059.stdout: "device_class_flags": {} 2020-03-11T22:34:23.206 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.206 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_metadata": [ 2020-03-11T22:34:23.206 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.206 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 0, 2020-03-11T22:34:23.206 INFO:tasks.workunit.client.0.smithi059.stdout: "arch": "x86_64", 2020-03-11T22:34:23.206 INFO:tasks.workunit.client.0.smithi059.stdout: "back_addr": "[v2:127.0.0.1:6804/23083,v1:127.0.0.1:6805/23083]", 2020-03-11T22:34:23.207 INFO:tasks.workunit.client.0.smithi059.stdout: "back_iface": "", 2020-03-11T22:34:23.207 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs": "1", 2020-03-11T22:34:23.207 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:34:23.207 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:34:23.207 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:34:23.207 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:34:23.208 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:34:23.208 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:34:23.208 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:34:23.208 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2020-03-11T22:34:23.208 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:34:23.208 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:34:23.208 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:34:23.209 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:34:23.209 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_release": "octopus", 2020-03-11T22:34:23.209 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:34:23.209 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:34:23.209 INFO:tasks.workunit.client.0.smithi059.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-03-11T22:34:23.209 INFO:tasks.workunit.client.0.smithi059.stdout: "default_device_class": "hdd", 2020-03-11T22:34:23.209 INFO:tasks.workunit.client.0.smithi059.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4J2P7", 2020-03-11T22:34:23.210 INFO:tasks.workunit.client.0.smithi059.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:34:23.210 INFO:tasks.workunit.client.0.smithi059.stdout: "devices": "sda", 2020-03-11T22:34:23.210 INFO:tasks.workunit.client.0.smithi059.stdout: "distro": "centos", 2020-03-11T22:34:23.210 INFO:tasks.workunit.client.0.smithi059.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:34:23.210 INFO:tasks.workunit.client.0.smithi059.stdout: "distro_version": "8", 2020-03-11T22:34:23.210 INFO:tasks.workunit.client.0.smithi059.stdout: "front_addr": "[v2:127.0.0.1:6802/23083,v1:127.0.0.1:6803/23083]", 2020-03-11T22:34:23.211 INFO:tasks.workunit.client.0.smithi059.stdout: "front_iface": "", 2020-03-11T22:34:23.211 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/23083,v1:127.0.0.1:6809/23083]", 2020-03-11T22:34:23.211 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/23083,v1:127.0.0.1:6807/23083]", 2020-03-11T22:34:23.211 INFO:tasks.workunit.client.0.smithi059.stdout: "hostname": "smithi059", 2020-03-11T22:34:23.211 INFO:tasks.workunit.client.0.smithi059.stdout: "journal_rotational": "1", 2020-03-11T22:34:23.211 INFO:tasks.workunit.client.0.smithi059.stdout: "kernel_description": "#1 SMP Wed Feb 5 02:00:39 UTC 2020", 2020-03-11T22:34:23.211 INFO:tasks.workunit.client.0.smithi059.stdout: "kernel_version": "4.18.0-147.5.1.el8_1.x86_64", 2020-03-11T22:34:23.211 INFO:tasks.workunit.client.0.smithi059.stdout: "mem_swap_kb": "0", 2020-03-11T22:34:23.212 INFO:tasks.workunit.client.0.smithi059.stdout: "mem_total_kb": "32662084", 2020-03-11T22:34:23.212 INFO:tasks.workunit.client.0.smithi059.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:34:23.212 INFO:tasks.workunit.client.0.smithi059.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:34:23.212 INFO:tasks.workunit.client.0.smithi059.stdout: "os": "Linux", 2020-03-11T22:34:23.212 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_data": "td/divergent-priors/0", 2020-03-11T22:34:23.212 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:34:23.212 INFO:tasks.workunit.client.0.smithi059.stdout: "rotational": "1" 2020-03-11T22:34:23.213 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.213 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.213 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 1, 2020-03-11T22:34:23.213 INFO:tasks.workunit.client.0.smithi059.stdout: "arch": "x86_64", 2020-03-11T22:34:23.213 INFO:tasks.workunit.client.0.smithi059.stdout: "back_addr": "[v2:127.0.0.1:6812/18723,v1:127.0.0.1:6813/18723]", 2020-03-11T22:34:23.213 INFO:tasks.workunit.client.0.smithi059.stdout: "back_iface": "", 2020-03-11T22:34:23.213 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs": "1", 2020-03-11T22:34:23.214 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:34:23.214 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:34:23.214 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:34:23.214 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:34:23.214 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:34:23.214 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:34:23.214 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:34:23.215 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2020-03-11T22:34:23.215 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:34:23.215 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:34:23.215 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:34:23.215 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:34:23.215 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_release": "octopus", 2020-03-11T22:34:23.215 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:34:23.215 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:34:23.216 INFO:tasks.workunit.client.0.smithi059.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-03-11T22:34:23.216 INFO:tasks.workunit.client.0.smithi059.stdout: "default_device_class": "hdd", 2020-03-11T22:34:23.216 INFO:tasks.workunit.client.0.smithi059.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4J2P7", 2020-03-11T22:34:23.216 INFO:tasks.workunit.client.0.smithi059.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:34:23.216 INFO:tasks.workunit.client.0.smithi059.stdout: "devices": "sda", 2020-03-11T22:34:23.216 INFO:tasks.workunit.client.0.smithi059.stdout: "distro": "centos", 2020-03-11T22:34:23.216 INFO:tasks.workunit.client.0.smithi059.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:34:23.217 INFO:tasks.workunit.client.0.smithi059.stdout: "distro_version": "8", 2020-03-11T22:34:23.217 INFO:tasks.workunit.client.0.smithi059.stdout: "front_addr": "[v2:127.0.0.1:6810/18723,v1:127.0.0.1:6811/18723]", 2020-03-11T22:34:23.217 INFO:tasks.workunit.client.0.smithi059.stdout: "front_iface": "", 2020-03-11T22:34:23.217 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/18723,v1:127.0.0.1:6817/18723]", 2020-03-11T22:34:23.217 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/18723,v1:127.0.0.1:6815/18723]", 2020-03-11T22:34:23.217 INFO:tasks.workunit.client.0.smithi059.stdout: "hostname": "smithi059", 2020-03-11T22:34:23.217 INFO:tasks.workunit.client.0.smithi059.stdout: "journal_rotational": "1", 2020-03-11T22:34:23.217 INFO:tasks.workunit.client.0.smithi059.stdout: "kernel_description": "#1 SMP Wed Feb 5 02:00:39 UTC 2020", 2020-03-11T22:34:23.218 INFO:tasks.workunit.client.0.smithi059.stdout: "kernel_version": "4.18.0-147.5.1.el8_1.x86_64", 2020-03-11T22:34:23.218 INFO:tasks.workunit.client.0.smithi059.stdout: "mem_swap_kb": "0", 2020-03-11T22:34:23.218 INFO:tasks.workunit.client.0.smithi059.stdout: "mem_total_kb": "32662084", 2020-03-11T22:34:23.218 INFO:tasks.workunit.client.0.smithi059.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:34:23.218 INFO:tasks.workunit.client.0.smithi059.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:34:23.218 INFO:tasks.workunit.client.0.smithi059.stdout: "os": "Linux", 2020-03-11T22:34:23.219 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_data": "td/divergent-priors/1", 2020-03-11T22:34:23.219 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:34:23.219 INFO:tasks.workunit.client.0.smithi059.stdout: "rotational": "1" 2020-03-11T22:34:23.219 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.219 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.219 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 2, 2020-03-11T22:34:23.220 INFO:tasks.workunit.client.0.smithi059.stdout: "arch": "x86_64", 2020-03-11T22:34:23.220 INFO:tasks.workunit.client.0.smithi059.stdout: "back_addr": "[v2:127.0.0.1:6812/23277,v1:127.0.0.1:6813/23277]", 2020-03-11T22:34:23.220 INFO:tasks.workunit.client.0.smithi059.stdout: "back_iface": "", 2020-03-11T22:34:23.220 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs": "1", 2020-03-11T22:34:23.220 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:34:23.220 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:34:23.220 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:34:23.220 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:34:23.221 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:34:23.221 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:34:23.221 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:34:23.221 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2020-03-11T22:34:23.221 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:34:23.221 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:34:23.221 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:34:23.222 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:34:23.222 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_release": "octopus", 2020-03-11T22:34:23.222 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:34:23.222 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:34:23.222 INFO:tasks.workunit.client.0.smithi059.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-03-11T22:34:23.222 INFO:tasks.workunit.client.0.smithi059.stdout: "default_device_class": "hdd", 2020-03-11T22:34:23.222 INFO:tasks.workunit.client.0.smithi059.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4J2P7", 2020-03-11T22:34:23.223 INFO:tasks.workunit.client.0.smithi059.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:34:23.223 INFO:tasks.workunit.client.0.smithi059.stdout: "devices": "sda", 2020-03-11T22:34:23.223 INFO:tasks.workunit.client.0.smithi059.stdout: "distro": "centos", 2020-03-11T22:34:23.223 INFO:tasks.workunit.client.0.smithi059.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:34:23.223 INFO:tasks.workunit.client.0.smithi059.stdout: "distro_version": "8", 2020-03-11T22:34:23.223 INFO:tasks.workunit.client.0.smithi059.stdout: "front_addr": "[v2:127.0.0.1:6810/23277,v1:127.0.0.1:6811/23277]", 2020-03-11T22:34:23.223 INFO:tasks.workunit.client.0.smithi059.stdout: "front_iface": "", 2020-03-11T22:34:23.223 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/23277,v1:127.0.0.1:6817/23277]", 2020-03-11T22:34:23.224 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/23277,v1:127.0.0.1:6815/23277]", 2020-03-11T22:34:23.224 INFO:tasks.workunit.client.0.smithi059.stdout: "hostname": "smithi059", 2020-03-11T22:34:23.224 INFO:tasks.workunit.client.0.smithi059.stdout: "journal_rotational": "1", 2020-03-11T22:34:23.224 INFO:tasks.workunit.client.0.smithi059.stdout: "kernel_description": "#1 SMP Wed Feb 5 02:00:39 UTC 2020", 2020-03-11T22:34:23.224 INFO:tasks.workunit.client.0.smithi059.stdout: "kernel_version": "4.18.0-147.5.1.el8_1.x86_64", 2020-03-11T22:34:23.224 INFO:tasks.workunit.client.0.smithi059.stdout: "mem_swap_kb": "0", 2020-03-11T22:34:23.224 INFO:tasks.workunit.client.0.smithi059.stdout: "mem_total_kb": "32662084", 2020-03-11T22:34:23.225 INFO:tasks.workunit.client.0.smithi059.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:34:23.225 INFO:tasks.workunit.client.0.smithi059.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:34:23.225 INFO:tasks.workunit.client.0.smithi059.stdout: "os": "Linux", 2020-03-11T22:34:23.225 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_data": "td/divergent-priors/2", 2020-03-11T22:34:23.225 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:34:23.225 INFO:tasks.workunit.client.0.smithi059.stdout: "rotational": "1" 2020-03-11T22:34:23.225 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:34:23.226 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:34:23.226 INFO:tasks.workunit.client.0.smithi059.stdout: "osdmap_first_committed": 1, 2020-03-11T22:34:23.226 INFO:tasks.workunit.client.0.smithi059.stdout: "osdmap_last_committed": 39, 2020-03-11T22:34:23.226 INFO:tasks.workunit.client.0.smithi059.stdout: "crushmap": { 2020-03-11T22:34:23.226 INFO:tasks.workunit.client.0.smithi059.stdout: "devices": [ 2020-03-11T22:34:23.226 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.226 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 0, 2020-03-11T22:34:23.227 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "osd.0", 2020-03-11T22:34:23.227 INFO:tasks.workunit.client.0.smithi059.stdout: "class": "hdd" 2020-03-11T22:34:23.227 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.227 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.227 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 1, 2020-03-11T22:34:23.227 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "osd.1", 2020-03-11T22:34:23.227 INFO:tasks.workunit.client.0.smithi059.stdout: "class": "hdd" 2020-03-11T22:34:23.227 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.228 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.228 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 2, 2020-03-11T22:34:23.228 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "osd.2", 2020-03-11T22:34:23.228 INFO:tasks.workunit.client.0.smithi059.stdout: "class": "hdd" 2020-03-11T22:34:23.228 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:34:23.228 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:34:23.228 INFO:tasks.workunit.client.0.smithi059.stdout: "types": [ 2020-03-11T22:34:23.229 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.229 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 0, 2020-03-11T22:34:23.229 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "osd" 2020-03-11T22:34:23.229 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.229 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.229 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 1, 2020-03-11T22:34:23.229 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "host" 2020-03-11T22:34:23.230 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.230 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.230 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 2, 2020-03-11T22:34:23.230 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "chassis" 2020-03-11T22:34:23.230 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.230 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.231 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 3, 2020-03-11T22:34:23.231 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "rack" 2020-03-11T22:34:23.231 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.231 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.231 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 4, 2020-03-11T22:34:23.231 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "row" 2020-03-11T22:34:23.231 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.232 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.232 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 5, 2020-03-11T22:34:23.232 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "pdu" 2020-03-11T22:34:23.232 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.232 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.232 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 6, 2020-03-11T22:34:23.232 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "pod" 2020-03-11T22:34:23.233 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.233 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.233 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 7, 2020-03-11T22:34:23.233 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "room" 2020-03-11T22:34:23.233 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.233 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.233 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 8, 2020-03-11T22:34:23.234 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "datacenter" 2020-03-11T22:34:23.234 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.234 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.234 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 9, 2020-03-11T22:34:23.234 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "zone" 2020-03-11T22:34:23.234 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.234 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.234 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 10, 2020-03-11T22:34:23.235 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "region" 2020-03-11T22:34:23.235 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.235 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.235 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 11, 2020-03-11T22:34:23.235 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "root" 2020-03-11T22:34:23.235 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:34:23.235 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:34:23.236 INFO:tasks.workunit.client.0.smithi059.stdout: "buckets": [ 2020-03-11T22:34:23.236 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.236 INFO:tasks.workunit.client.0.smithi059.stdout: "id": -1, 2020-03-11T22:34:23.236 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "default", 2020-03-11T22:34:23.236 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 11, 2020-03-11T22:34:23.236 INFO:tasks.workunit.client.0.smithi059.stdout: "type_name": "root", 2020-03-11T22:34:23.236 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 19206, 2020-03-11T22:34:23.237 INFO:tasks.workunit.client.0.smithi059.stdout: "alg": "straw2", 2020-03-11T22:34:23.237 INFO:tasks.workunit.client.0.smithi059.stdout: "hash": "rjenkins1", 2020-03-11T22:34:23.237 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2020-03-11T22:34:23.237 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.237 INFO:tasks.workunit.client.0.smithi059.stdout: "id": -3, 2020-03-11T22:34:23.237 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 19206, 2020-03-11T22:34:23.237 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 0 2020-03-11T22:34:23.238 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:34:23.238 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:34:23.238 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.238 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.238 INFO:tasks.workunit.client.0.smithi059.stdout: "id": -2, 2020-03-11T22:34:23.238 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "default~hdd", 2020-03-11T22:34:23.238 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 11, 2020-03-11T22:34:23.239 INFO:tasks.workunit.client.0.smithi059.stdout: "type_name": "root", 2020-03-11T22:34:23.239 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 19206, 2020-03-11T22:34:23.239 INFO:tasks.workunit.client.0.smithi059.stdout: "alg": "straw2", 2020-03-11T22:34:23.239 INFO:tasks.workunit.client.0.smithi059.stdout: "hash": "rjenkins1", 2020-03-11T22:34:23.239 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2020-03-11T22:34:23.239 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.239 INFO:tasks.workunit.client.0.smithi059.stdout: "id": -4, 2020-03-11T22:34:23.239 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 19206, 2020-03-11T22:34:23.240 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 0 2020-03-11T22:34:23.240 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:34:23.240 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:34:23.240 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.240 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.240 INFO:tasks.workunit.client.0.smithi059.stdout: "id": -3, 2020-03-11T22:34:23.240 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "smithi059", 2020-03-11T22:34:23.241 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 1, 2020-03-11T22:34:23.241 INFO:tasks.workunit.client.0.smithi059.stdout: "type_name": "host", 2020-03-11T22:34:23.241 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 19206, 2020-03-11T22:34:23.241 INFO:tasks.workunit.client.0.smithi059.stdout: "alg": "straw2", 2020-03-11T22:34:23.241 INFO:tasks.workunit.client.0.smithi059.stdout: "hash": "rjenkins1", 2020-03-11T22:34:23.241 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2020-03-11T22:34:23.241 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.242 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 0, 2020-03-11T22:34:23.242 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 6402, 2020-03-11T22:34:23.242 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 0 2020-03-11T22:34:23.242 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.242 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.242 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 1, 2020-03-11T22:34:23.242 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 6402, 2020-03-11T22:34:23.243 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 1 2020-03-11T22:34:23.243 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.243 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.243 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 2, 2020-03-11T22:34:23.243 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 6402, 2020-03-11T22:34:23.243 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 2 2020-03-11T22:34:23.243 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:34:23.243 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:34:23.244 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.244 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.244 INFO:tasks.workunit.client.0.smithi059.stdout: "id": -4, 2020-03-11T22:34:23.244 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "smithi059~hdd", 2020-03-11T22:34:23.244 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 1, 2020-03-11T22:34:23.244 INFO:tasks.workunit.client.0.smithi059.stdout: "type_name": "host", 2020-03-11T22:34:23.244 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 19206, 2020-03-11T22:34:23.245 INFO:tasks.workunit.client.0.smithi059.stdout: "alg": "straw2", 2020-03-11T22:34:23.245 INFO:tasks.workunit.client.0.smithi059.stdout: "hash": "rjenkins1", 2020-03-11T22:34:23.245 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2020-03-11T22:34:23.245 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.245 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 0, 2020-03-11T22:34:23.245 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 6402, 2020-03-11T22:34:23.245 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 0 2020-03-11T22:34:23.246 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.246 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.246 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 1, 2020-03-11T22:34:23.246 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 6402, 2020-03-11T22:34:23.246 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 1 2020-03-11T22:34:23.246 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.246 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.247 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 2, 2020-03-11T22:34:23.247 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 6402, 2020-03-11T22:34:23.247 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 2 2020-03-11T22:34:23.247 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:34:23.247 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:34:23.247 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:34:23.247 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:34:23.247 INFO:tasks.workunit.client.0.smithi059.stdout: "rules": [ 2020-03-11T22:34:23.248 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.248 INFO:tasks.workunit.client.0.smithi059.stdout: "rule_id": 0, 2020-03-11T22:34:23.248 INFO:tasks.workunit.client.0.smithi059.stdout: "rule_name": "replicated_rule", 2020-03-11T22:34:23.248 INFO:tasks.workunit.client.0.smithi059.stdout: "ruleset": 0, 2020-03-11T22:34:23.248 INFO:tasks.workunit.client.0.smithi059.stdout: "type": 1, 2020-03-11T22:34:23.248 INFO:tasks.workunit.client.0.smithi059.stdout: "min_size": 1, 2020-03-11T22:34:23.248 INFO:tasks.workunit.client.0.smithi059.stdout: "max_size": 10, 2020-03-11T22:34:23.249 INFO:tasks.workunit.client.0.smithi059.stdout: "steps": [ 2020-03-11T22:34:23.249 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.249 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "take", 2020-03-11T22:34:23.249 INFO:tasks.workunit.client.0.smithi059.stdout: "item": -1, 2020-03-11T22:34:23.249 INFO:tasks.workunit.client.0.smithi059.stdout: "item_name": "default" 2020-03-11T22:34:23.249 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.249 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.250 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "choose_firstn", 2020-03-11T22:34:23.250 INFO:tasks.workunit.client.0.smithi059.stdout: "num": 0, 2020-03-11T22:34:23.250 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "osd" 2020-03-11T22:34:23.250 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.250 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.250 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "emit" 2020-03-11T22:34:23.250 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:34:23.250 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:34:23.251 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:34:23.251 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:34:23.251 INFO:tasks.workunit.client.0.smithi059.stdout: "tunables": { 2020-03-11T22:34:23.251 INFO:tasks.workunit.client.0.smithi059.stdout: "choose_local_tries": 0, 2020-03-11T22:34:23.251 INFO:tasks.workunit.client.0.smithi059.stdout: "choose_local_fallback_tries": 0, 2020-03-11T22:34:23.251 INFO:tasks.workunit.client.0.smithi059.stdout: "choose_total_tries": 50, 2020-03-11T22:34:23.251 INFO:tasks.workunit.client.0.smithi059.stdout: "chooseleaf_descend_once": 1, 2020-03-11T22:34:23.252 INFO:tasks.workunit.client.0.smithi059.stdout: "chooseleaf_vary_r": 1, 2020-03-11T22:34:23.252 INFO:tasks.workunit.client.0.smithi059.stdout: "chooseleaf_stable": 1, 2020-03-11T22:34:23.252 INFO:tasks.workunit.client.0.smithi059.stdout: "straw_calc_version": 1, 2020-03-11T22:34:23.252 INFO:tasks.workunit.client.0.smithi059.stdout: "allowed_bucket_algs": 54, 2020-03-11T22:34:23.252 INFO:tasks.workunit.client.0.smithi059.stdout: "profile": "jewel", 2020-03-11T22:34:23.252 INFO:tasks.workunit.client.0.smithi059.stdout: "optimal_tunables": 1, 2020-03-11T22:34:23.252 INFO:tasks.workunit.client.0.smithi059.stdout: "legacy_tunables": 0, 2020-03-11T22:34:23.252 INFO:tasks.workunit.client.0.smithi059.stdout: "minimum_required_version": "jewel", 2020-03-11T22:34:23.253 INFO:tasks.workunit.client.0.smithi059.stdout: "require_feature_tunables": 1, 2020-03-11T22:34:23.253 INFO:tasks.workunit.client.0.smithi059.stdout: "require_feature_tunables2": 1, 2020-03-11T22:34:23.253 INFO:tasks.workunit.client.0.smithi059.stdout: "has_v2_rules": 0, 2020-03-11T22:34:23.253 INFO:tasks.workunit.client.0.smithi059.stdout: "require_feature_tunables3": 1, 2020-03-11T22:34:23.253 INFO:tasks.workunit.client.0.smithi059.stdout: "has_v3_rules": 0, 2020-03-11T22:34:23.253 INFO:tasks.workunit.client.0.smithi059.stdout: "has_v4_buckets": 1, 2020-03-11T22:34:23.253 INFO:tasks.workunit.client.0.smithi059.stdout: "require_feature_tunables5": 1, 2020-03-11T22:34:23.254 INFO:tasks.workunit.client.0.smithi059.stdout: "has_v5_rules": 0 2020-03-11T22:34:23.254 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.254 INFO:tasks.workunit.client.0.smithi059.stdout: "choose_args": {} 2020-03-11T22:34:23.254 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.254 INFO:tasks.workunit.client.0.smithi059.stdout: "fsmap": { 2020-03-11T22:34:23.254 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": 1, 2020-03-11T22:34:23.254 INFO:tasks.workunit.client.0.smithi059.stdout: "default_fscid": -1, 2020-03-11T22:34:23.255 INFO:tasks.workunit.client.0.smithi059.stdout: "compat": { 2020-03-11T22:34:23.255 INFO:tasks.workunit.client.0.smithi059.stdout: "compat": {}, 2020-03-11T22:34:23.255 INFO:tasks.workunit.client.0.smithi059.stdout: "ro_compat": {}, 2020-03-11T22:34:23.255 INFO:tasks.workunit.client.0.smithi059.stdout: "incompat": { 2020-03-11T22:34:23.255 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_1": "base v0.20", 2020-03-11T22:34:23.255 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_2": "client writeable ranges", 2020-03-11T22:34:23.255 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_3": "default file layouts on dirs", 2020-03-11T22:34:23.255 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_4": "dir inode in separate object", 2020-03-11T22:34:23.256 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_5": "mds uses versioned encoding", 2020-03-11T22:34:23.256 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_6": "dirfrag is stored in omap", 2020-03-11T22:34:23.256 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_8": "no anchor table", 2020-03-11T22:34:23.256 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_9": "file layout v2", 2020-03-11T22:34:23.256 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_10": "snaprealm v2" 2020-03-11T22:34:23.256 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:34:23.256 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.257 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_flags": { 2020-03-11T22:34:23.257 INFO:tasks.workunit.client.0.smithi059.stdout: "enable_multiple": false, 2020-03-11T22:34:23.257 INFO:tasks.workunit.client.0.smithi059.stdout: "ever_enabled_multiple": false 2020-03-11T22:34:23.257 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.257 INFO:tasks.workunit.client.0.smithi059.stdout: "standbys": [], 2020-03-11T22:34:23.257 INFO:tasks.workunit.client.0.smithi059.stdout: "filesystems": [] 2020-03-11T22:34:23.257 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.258 INFO:tasks.workunit.client.0.smithi059.stdout: "mdsmap_first_committed": 1, 2020-03-11T22:34:23.258 INFO:tasks.workunit.client.0.smithi059.stdout: "mdsmap_last_committed": 1, 2020-03-11T22:34:23.258 INFO:tasks.workunit.client.0.smithi059.stdout: "auth": { 2020-03-11T22:34:23.258 INFO:tasks.workunit.client.0.smithi059.stdout: "first_committed": 1, 2020-03-11T22:34:23.258 INFO:tasks.workunit.client.0.smithi059.stdout: "last_committed": 5, 2020-03-11T22:34:23.258 INFO:tasks.workunit.client.0.smithi059.stdout: "num_secrets": 3 2020-03-11T22:34:23.258 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.258 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pg": 1, 2020-03-11T22:34:23.259 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pg_active": 1, 2020-03-11T22:34:23.259 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pg_unknown": 0, 2020-03-11T22:34:23.259 INFO:tasks.workunit.client.0.smithi059.stdout: "num_osd": 3, 2020-03-11T22:34:23.259 INFO:tasks.workunit.client.0.smithi059.stdout: "pool_sum": { 2020-03-11T22:34:23.259 INFO:tasks.workunit.client.0.smithi059.stdout: "stat_sum": { 2020-03-11T22:34:23.259 INFO:tasks.workunit.client.0.smithi059.stdout: "num_bytes": 48300, 2020-03-11T22:34:23.259 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects": 100, 2020-03-11T22:34:23.260 INFO:tasks.workunit.client.0.smithi059.stdout: "num_object_clones": 0, 2020-03-11T22:34:23.260 INFO:tasks.workunit.client.0.smithi059.stdout: "num_object_copies": 300, 2020-03-11T22:34:23.260 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:34:23.260 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_missing": 0, 2020-03-11T22:34:23.260 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_degraded": 100, 2020-03-11T22:34:23.260 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_misplaced": 0, 2020-03-11T22:34:23.260 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_unfound": 0, 2020-03-11T22:34:23.261 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_dirty": 100, 2020-03-11T22:34:23.261 INFO:tasks.workunit.client.0.smithi059.stdout: "num_whiteouts": 0, 2020-03-11T22:34:23.261 INFO:tasks.workunit.client.0.smithi059.stdout: "num_read": 0, 2020-03-11T22:34:23.261 INFO:tasks.workunit.client.0.smithi059.stdout: "num_read_kb": 0, 2020-03-11T22:34:23.261 INFO:tasks.workunit.client.0.smithi059.stdout: "num_write": 100, 2020-03-11T22:34:23.261 INFO:tasks.workunit.client.0.smithi059.stdout: "num_write_kb": 100, 2020-03-11T22:34:23.261 INFO:tasks.workunit.client.0.smithi059.stdout: "num_scrub_errors": 0, 2020-03-11T22:34:23.261 INFO:tasks.workunit.client.0.smithi059.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:34:23.262 INFO:tasks.workunit.client.0.smithi059.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:34:23.262 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_recovered": 0, 2020-03-11T22:34:23.262 INFO:tasks.workunit.client.0.smithi059.stdout: "num_bytes_recovered": 0, 2020-03-11T22:34:23.262 INFO:tasks.workunit.client.0.smithi059.stdout: "num_keys_recovered": 0, 2020-03-11T22:34:23.262 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_omap": 0, 2020-03-11T22:34:23.262 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:34:23.262 INFO:tasks.workunit.client.0.smithi059.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:34:23.263 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush": 0, 2020-03-11T22:34:23.263 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush_kb": 0, 2020-03-11T22:34:23.263 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict": 0, 2020-03-11T22:34:23.263 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict_kb": 0, 2020-03-11T22:34:23.263 INFO:tasks.workunit.client.0.smithi059.stdout: "num_promote": 0, 2020-03-11T22:34:23.263 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush_mode_high": 0, 2020-03-11T22:34:23.263 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush_mode_low": 0, 2020-03-11T22:34:23.264 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict_mode_some": 0, 2020-03-11T22:34:23.264 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict_mode_full": 0, 2020-03-11T22:34:23.264 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_pinned": 0, 2020-03-11T22:34:23.264 INFO:tasks.workunit.client.0.smithi059.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:34:23.264 INFO:tasks.workunit.client.0.smithi059.stdout: "num_large_omap_objects": 0, 2020-03-11T22:34:23.264 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_manifest": 0, 2020-03-11T22:34:23.264 INFO:tasks.workunit.client.0.smithi059.stdout: "num_omap_bytes": 0, 2020-03-11T22:34:23.264 INFO:tasks.workunit.client.0.smithi059.stdout: "num_omap_keys": 0, 2020-03-11T22:34:23.265 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_repaired": 0 2020-03-11T22:34:23.265 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.265 INFO:tasks.workunit.client.0.smithi059.stdout: "store_stats": { 2020-03-11T22:34:23.265 INFO:tasks.workunit.client.0.smithi059.stdout: "total": 0, 2020-03-11T22:34:23.265 INFO:tasks.workunit.client.0.smithi059.stdout: "available": 0, 2020-03-11T22:34:23.266 INFO:tasks.workunit.client.0.smithi059.stdout: "internally_reserved": 0, 2020-03-11T22:34:23.266 INFO:tasks.workunit.client.0.smithi059.stdout: "allocated": 0, 2020-03-11T22:34:23.266 INFO:tasks.workunit.client.0.smithi059.stdout: "data_stored": 0, 2020-03-11T22:34:23.266 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed": 0, 2020-03-11T22:34:23.266 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_allocated": 0, 2020-03-11T22:34:23.266 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_original": 0, 2020-03-11T22:34:23.266 INFO:tasks.workunit.client.0.smithi059.stdout: "omap_allocated": 0, 2020-03-11T22:34:23.266 INFO:tasks.workunit.client.0.smithi059.stdout: "internal_metadata": 0 2020-03-11T22:34:23.267 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.267 INFO:tasks.workunit.client.0.smithi059.stdout: "log_size": 16, 2020-03-11T22:34:23.267 INFO:tasks.workunit.client.0.smithi059.stdout: "ondisk_log_size": 16, 2020-03-11T22:34:23.267 INFO:tasks.workunit.client.0.smithi059.stdout: "up": 2, 2020-03-11T22:34:23.267 INFO:tasks.workunit.client.0.smithi059.stdout: "acting": 2, 2020-03-11T22:34:23.267 INFO:tasks.workunit.client.0.smithi059.stdout: "num_store_stats": 0 2020-03-11T22:34:23.267 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.268 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_sum": { 2020-03-11T22:34:23.268 INFO:tasks.workunit.client.0.smithi059.stdout: "up_from": 0, 2020-03-11T22:34:23.268 INFO:tasks.workunit.client.0.smithi059.stdout: "seq": 0, 2020-03-11T22:34:23.268 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pgs": 3, 2020-03-11T22:34:23.268 INFO:tasks.workunit.client.0.smithi059.stdout: "num_osds": 3, 2020-03-11T22:34:23.268 INFO:tasks.workunit.client.0.smithi059.stdout: "num_per_pool_osds": 3, 2020-03-11T22:34:23.268 INFO:tasks.workunit.client.0.smithi059.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:34:23.269 INFO:tasks.workunit.client.0.smithi059.stdout: "kb": 314572800, 2020-03-11T22:34:23.269 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used": 3179264, 2020-03-11T22:34:23.269 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used_data": 33280, 2020-03-11T22:34:23.269 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used_omap": 13, 2020-03-11T22:34:23.269 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used_meta": 3145714, 2020-03-11T22:34:23.269 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_avail": 311393536, 2020-03-11T22:34:23.270 INFO:tasks.workunit.client.0.smithi059.stdout: "statfs": { 2020-03-11T22:34:23.270 INFO:tasks.workunit.client.0.smithi059.stdout: "total": 322122547200, 2020-03-11T22:34:23.270 INFO:tasks.workunit.client.0.smithi059.stdout: "available": 318866980864, 2020-03-11T22:34:23.270 INFO:tasks.workunit.client.0.smithi059.stdout: "internally_reserved": 3221225472, 2020-03-11T22:34:23.270 INFO:tasks.workunit.client.0.smithi059.stdout: "allocated": 34078720, 2020-03-11T22:34:23.270 INFO:tasks.workunit.client.0.smithi059.stdout: "data_stored": 404900, 2020-03-11T22:34:23.270 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed": 0, 2020-03-11T22:34:23.271 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_allocated": 0, 2020-03-11T22:34:23.271 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_original": 0, 2020-03-11T22:34:23.271 INFO:tasks.workunit.client.0.smithi059.stdout: "omap_allocated": 13558, 2020-03-11T22:34:23.271 INFO:tasks.workunit.client.0.smithi059.stdout: "internal_metadata": 3221211914 2020-03-11T22:34:23.271 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.271 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_peers": [], 2020-03-11T22:34:23.271 INFO:tasks.workunit.client.0.smithi059.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:34:23.272 INFO:tasks.workunit.client.0.smithi059.stdout: "num_snap_trimming": 0, 2020-03-11T22:34:23.272 INFO:tasks.workunit.client.0.smithi059.stdout: "num_shards_repaired": 1, 2020-03-11T22:34:23.272 INFO:tasks.workunit.client.0.smithi059.stdout: "op_queue_age_hist": { 2020-03-11T22:34:23.272 INFO:tasks.workunit.client.0.smithi059.stdout: "histogram": [], 2020-03-11T22:34:23.272 INFO:tasks.workunit.client.0.smithi059.stdout: "upper_bound": 1 2020-03-11T22:34:23.272 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.272 INFO:tasks.workunit.client.0.smithi059.stdout: "perf_stat": { 2020-03-11T22:34:23.272 INFO:tasks.workunit.client.0.smithi059.stdout: "commit_latency_ms": 0, 2020-03-11T22:34:23.273 INFO:tasks.workunit.client.0.smithi059.stdout: "apply_latency_ms": 0, 2020-03-11T22:34:23.273 INFO:tasks.workunit.client.0.smithi059.stdout: "commit_latency_ns": 0, 2020-03-11T22:34:23.273 INFO:tasks.workunit.client.0.smithi059.stdout: "apply_latency_ns": 0 2020-03-11T22:34:23.273 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.273 INFO:tasks.workunit.client.0.smithi059.stdout: "alerts": [], 2020-03-11T22:34:23.273 INFO:tasks.workunit.client.0.smithi059.stdout: "network_ping_times": [] 2020-03-11T22:34:23.273 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.274 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_sum_by_class": { 2020-03-11T22:34:23.274 INFO:tasks.workunit.client.0.smithi059.stdout: "hdd": { 2020-03-11T22:34:23.274 INFO:tasks.workunit.client.0.smithi059.stdout: "up_from": 0, 2020-03-11T22:34:23.274 INFO:tasks.workunit.client.0.smithi059.stdout: "seq": 0, 2020-03-11T22:34:23.274 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pgs": 3, 2020-03-11T22:34:23.274 INFO:tasks.workunit.client.0.smithi059.stdout: "num_osds": 3, 2020-03-11T22:34:23.274 INFO:tasks.workunit.client.0.smithi059.stdout: "num_per_pool_osds": 3, 2020-03-11T22:34:23.275 INFO:tasks.workunit.client.0.smithi059.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:34:23.275 INFO:tasks.workunit.client.0.smithi059.stdout: "kb": 314572800, 2020-03-11T22:34:23.275 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used": 3179264, 2020-03-11T22:34:23.275 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used_data": 33280, 2020-03-11T22:34:23.275 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used_omap": 13, 2020-03-11T22:34:23.275 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used_meta": 3145714, 2020-03-11T22:34:23.275 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_avail": 311393536, 2020-03-11T22:34:23.275 INFO:tasks.workunit.client.0.smithi059.stdout: "statfs": { 2020-03-11T22:34:23.276 INFO:tasks.workunit.client.0.smithi059.stdout: "total": 322122547200, 2020-03-11T22:34:23.276 INFO:tasks.workunit.client.0.smithi059.stdout: "available": 318866980864, 2020-03-11T22:34:23.276 INFO:tasks.workunit.client.0.smithi059.stdout: "internally_reserved": 3221225472, 2020-03-11T22:34:23.276 INFO:tasks.workunit.client.0.smithi059.stdout: "allocated": 34078720, 2020-03-11T22:34:23.276 INFO:tasks.workunit.client.0.smithi059.stdout: "data_stored": 404900, 2020-03-11T22:34:23.276 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed": 0, 2020-03-11T22:34:23.276 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_allocated": 0, 2020-03-11T22:34:23.277 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_original": 0, 2020-03-11T22:34:23.277 INFO:tasks.workunit.client.0.smithi059.stdout: "omap_allocated": 13558, 2020-03-11T22:34:23.277 INFO:tasks.workunit.client.0.smithi059.stdout: "internal_metadata": 3221211914 2020-03-11T22:34:23.277 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.277 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_peers": [], 2020-03-11T22:34:23.277 INFO:tasks.workunit.client.0.smithi059.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:34:23.277 INFO:tasks.workunit.client.0.smithi059.stdout: "num_snap_trimming": 0, 2020-03-11T22:34:23.278 INFO:tasks.workunit.client.0.smithi059.stdout: "num_shards_repaired": 1, 2020-03-11T22:34:23.278 INFO:tasks.workunit.client.0.smithi059.stdout: "op_queue_age_hist": { 2020-03-11T22:34:23.278 INFO:tasks.workunit.client.0.smithi059.stdout: "histogram": [], 2020-03-11T22:34:23.278 INFO:tasks.workunit.client.0.smithi059.stdout: "upper_bound": 1 2020-03-11T22:34:23.278 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.278 INFO:tasks.workunit.client.0.smithi059.stdout: "perf_stat": { 2020-03-11T22:34:23.278 INFO:tasks.workunit.client.0.smithi059.stdout: "commit_latency_ms": 0, 2020-03-11T22:34:23.278 INFO:tasks.workunit.client.0.smithi059.stdout: "apply_latency_ms": 0, 2020-03-11T22:34:23.279 INFO:tasks.workunit.client.0.smithi059.stdout: "commit_latency_ns": 0, 2020-03-11T22:34:23.279 INFO:tasks.workunit.client.0.smithi059.stdout: "apply_latency_ns": 0 2020-03-11T22:34:23.279 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.279 INFO:tasks.workunit.client.0.smithi059.stdout: "alerts": [], 2020-03-11T22:34:23.279 INFO:tasks.workunit.client.0.smithi059.stdout: "network_ping_times": [] 2020-03-11T22:34:23.279 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:34:23.279 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.280 INFO:tasks.workunit.client.0.smithi059.stdout: "pool_stats": [ 2020-03-11T22:34:23.280 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.280 INFO:tasks.workunit.client.0.smithi059.stdout: "poolid": 1, 2020-03-11T22:34:23.280 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pg": 1, 2020-03-11T22:34:23.280 INFO:tasks.workunit.client.0.smithi059.stdout: "stat_sum": { 2020-03-11T22:34:23.280 INFO:tasks.workunit.client.0.smithi059.stdout: "num_bytes": 48300, 2020-03-11T22:34:23.280 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects": 100, 2020-03-11T22:34:23.281 INFO:tasks.workunit.client.0.smithi059.stdout: "num_object_clones": 0, 2020-03-11T22:34:23.281 INFO:tasks.workunit.client.0.smithi059.stdout: "num_object_copies": 300, 2020-03-11T22:34:23.281 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:34:23.281 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_missing": 0, 2020-03-11T22:34:23.281 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_degraded": 100, 2020-03-11T22:34:23.281 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_misplaced": 0, 2020-03-11T22:34:23.281 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_unfound": 0, 2020-03-11T22:34:23.282 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_dirty": 100, 2020-03-11T22:34:23.282 INFO:tasks.workunit.client.0.smithi059.stdout: "num_whiteouts": 0, 2020-03-11T22:34:23.282 INFO:tasks.workunit.client.0.smithi059.stdout: "num_read": 0, 2020-03-11T22:34:23.282 INFO:tasks.workunit.client.0.smithi059.stdout: "num_read_kb": 0, 2020-03-11T22:34:23.282 INFO:tasks.workunit.client.0.smithi059.stdout: "num_write": 100, 2020-03-11T22:34:23.282 INFO:tasks.workunit.client.0.smithi059.stdout: "num_write_kb": 100, 2020-03-11T22:34:23.282 INFO:tasks.workunit.client.0.smithi059.stdout: "num_scrub_errors": 0, 2020-03-11T22:34:23.283 INFO:tasks.workunit.client.0.smithi059.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:34:23.283 INFO:tasks.workunit.client.0.smithi059.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:34:23.283 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_recovered": 0, 2020-03-11T22:34:23.283 INFO:tasks.workunit.client.0.smithi059.stdout: "num_bytes_recovered": 0, 2020-03-11T22:34:23.283 INFO:tasks.workunit.client.0.smithi059.stdout: "num_keys_recovered": 0, 2020-03-11T22:34:23.283 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_omap": 0, 2020-03-11T22:34:23.283 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:34:23.284 INFO:tasks.workunit.client.0.smithi059.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:34:23.284 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush": 0, 2020-03-11T22:34:23.284 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush_kb": 0, 2020-03-11T22:34:23.284 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict": 0, 2020-03-11T22:34:23.284 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict_kb": 0, 2020-03-11T22:34:23.284 INFO:tasks.workunit.client.0.smithi059.stdout: "num_promote": 0, 2020-03-11T22:34:23.284 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush_mode_high": 0, 2020-03-11T22:34:23.284 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush_mode_low": 0, 2020-03-11T22:34:23.285 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict_mode_some": 0, 2020-03-11T22:34:23.285 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict_mode_full": 0, 2020-03-11T22:34:23.285 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_pinned": 0, 2020-03-11T22:34:23.285 INFO:tasks.workunit.client.0.smithi059.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:34:23.285 INFO:tasks.workunit.client.0.smithi059.stdout: "num_large_omap_objects": 0, 2020-03-11T22:34:23.285 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_manifest": 0, 2020-03-11T22:34:23.285 INFO:tasks.workunit.client.0.smithi059.stdout: "num_omap_bytes": 0, 2020-03-11T22:34:23.286 INFO:tasks.workunit.client.0.smithi059.stdout: "num_omap_keys": 0, 2020-03-11T22:34:23.286 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_repaired": 0 2020-03-11T22:34:23.286 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.286 INFO:tasks.workunit.client.0.smithi059.stdout: "store_stats": { 2020-03-11T22:34:23.286 INFO:tasks.workunit.client.0.smithi059.stdout: "total": 0, 2020-03-11T22:34:23.286 INFO:tasks.workunit.client.0.smithi059.stdout: "available": 0, 2020-03-11T22:34:23.286 INFO:tasks.workunit.client.0.smithi059.stdout: "internally_reserved": 0, 2020-03-11T22:34:23.287 INFO:tasks.workunit.client.0.smithi059.stdout: "allocated": 19660800, 2020-03-11T22:34:23.287 INFO:tasks.workunit.client.0.smithi059.stdout: "data_stored": 144900, 2020-03-11T22:34:23.287 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed": 0, 2020-03-11T22:34:23.287 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_allocated": 0, 2020-03-11T22:34:23.287 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_original": 0, 2020-03-11T22:34:23.287 INFO:tasks.workunit.client.0.smithi059.stdout: "omap_allocated": 0, 2020-03-11T22:34:23.287 INFO:tasks.workunit.client.0.smithi059.stdout: "internal_metadata": 0 2020-03-11T22:34:23.287 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.288 INFO:tasks.workunit.client.0.smithi059.stdout: "log_size": 16, 2020-03-11T22:34:23.288 INFO:tasks.workunit.client.0.smithi059.stdout: "ondisk_log_size": 16, 2020-03-11T22:34:23.288 INFO:tasks.workunit.client.0.smithi059.stdout: "up": 2, 2020-03-11T22:34:23.288 INFO:tasks.workunit.client.0.smithi059.stdout: "acting": 2, 2020-03-11T22:34:23.288 INFO:tasks.workunit.client.0.smithi059.stdout: "num_store_stats": 3 2020-03-11T22:34:23.288 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:34:23.288 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:34:23.289 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_stats": [ 2020-03-11T22:34:23.289 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.289 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 0, 2020-03-11T22:34:23.289 INFO:tasks.workunit.client.0.smithi059.stdout: "seq": 150323855370 2020-03-11T22:34:23.289 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.289 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.289 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 1, 2020-03-11T22:34:23.290 INFO:tasks.workunit.client.0.smithi059.stdout: "seq": 42949672975 2020-03-11T22:34:23.290 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.290 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.290 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 2, 2020-03-11T22:34:23.290 INFO:tasks.workunit.client.0.smithi059.stdout: "seq": 154618822666 2020-03-11T22:34:23.290 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:34:23.290 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:34:23.290 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pg_by_state": [ 2020-03-11T22:34:23.291 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.291 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "active+undersized+degraded", 2020-03-11T22:34:23.291 INFO:tasks.workunit.client.0.smithi059.stdout: "num": 1 2020-03-11T22:34:23.291 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:34:23.291 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:34:23.291 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pg_by_osd": [ 2020-03-11T22:34:23.291 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.292 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 2, 2020-03-11T22:34:23.292 INFO:tasks.workunit.client.0.smithi059.stdout: "num_primary_pg": 0, 2020-03-11T22:34:23.292 INFO:tasks.workunit.client.0.smithi059.stdout: "num_acting_pg": 1, 2020-03-11T22:34:23.292 INFO:tasks.workunit.client.0.smithi059.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:34:23.292 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.292 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.292 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 1, 2020-03-11T22:34:23.292 INFO:tasks.workunit.client.0.smithi059.stdout: "num_primary_pg": 0, 2020-03-11T22:34:23.293 INFO:tasks.workunit.client.0.smithi059.stdout: "num_acting_pg": 0, 2020-03-11T22:34:23.293 INFO:tasks.workunit.client.0.smithi059.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:34:23.293 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.293 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.293 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 0, 2020-03-11T22:34:23.294 INFO:tasks.workunit.client.0.smithi059.stdout: "num_primary_pg": 1, 2020-03-11T22:34:23.294 INFO:tasks.workunit.client.0.smithi059.stdout: "num_acting_pg": 1, 2020-03-11T22:34:23.294 INFO:tasks.workunit.client.0.smithi059.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:34:23.294 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:34:23.294 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:34:23.294 INFO:tasks.workunit.client.0.smithi059.stdout: "purged_snaps": [ 2020-03-11T22:34:23.294 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:34:23.295 INFO:tasks.workunit.client.0.smithi059.stdout: "pool": 1, 2020-03-11T22:34:23.295 INFO:tasks.workunit.client.0.smithi059.stdout: "purged_snaps": {} 2020-03-11T22:34:23.295 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:34:23.295 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:34:23.295 INFO:tasks.workunit.client.0.smithi059.stdout: "servicemap": { 2020-03-11T22:34:23.295 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": 1, 2020-03-11T22:34:23.295 INFO:tasks.workunit.client.0.smithi059.stdout: "modified": "2020-03-11T22:32:42.368687+0000", 2020-03-11T22:34:23.295 INFO:tasks.workunit.client.0.smithi059.stdout: "services": {} 2020-03-11T22:34:23.296 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:34:23.296 INFO:tasks.workunit.client.0.smithi059.stdout: "paxos": { 2020-03-11T22:34:23.296 INFO:tasks.workunit.client.0.smithi059.stdout: "first_committed": 1, 2020-03-11T22:34:23.296 INFO:tasks.workunit.client.0.smithi059.stdout: "last_committed": 306, 2020-03-11T22:34:23.296 INFO:tasks.workunit.client.0.smithi059.stdout: "last_pn": 0, 2020-03-11T22:34:23.296 INFO:tasks.workunit.client.0.smithi059.stdout: "accepted_pn": 0 2020-03-11T22:34:23.296 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:34:23.297 INFO:tasks.workunit.client.0.smithi059.stdout:} 2020-03-11T22:34:23.298 INFO:tasks.workunit.client.0.smithi059.stderr:report 1505181949 2020-03-11T22:34:23.298 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: return 1 2020-03-11T22:34:23.298 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:153: TEST_divergent: expr 2 + 2 2020-03-11T22:34:23.298 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:153: TEST_divergent: objname=existing_4 2020-03-11T22:34:23.298 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:154: TEST_divergent: echo 'writing non-divergent object existing_4' 2020-03-11T22:34:23.298 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:155: TEST_divergent: ceph pg dump pgs 2020-03-11T22:34:23.299 INFO:tasks.workunit.client.0.smithi059.stdout:writing non-divergent object existing_4 2020-03-11T22:34:23.484 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:34:23.484 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 100 0 100 0 0 48300 0 0 16 16 active+undersized+degraded 2020-03-11T22:33:47.576341+0000 29'100 39:119 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:33:04.364850+0000 0'0 2020-03-11T22:33:04.364850+0000 0 2020-03-11T22:34:23.484 INFO:tasks.workunit.client.0.smithi059.stdout: 2020-03-11T22:34:23.484 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:34:23.486 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2020-03-11T22:34:23.493 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:156: TEST_divergent: rados -p test put existing_4 /etc/resolv.conf 2020-03-11T22:34:23.570 INFO:tasks.workunit.client.0.smithi059.stdout:delay recovery 2020-03-11T22:34:23.570 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:159: TEST_divergent: echo 'delay recovery' 2020-03-11T22:34:23.570 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:160: TEST_divergent: ceph pg dump pgs 2020-03-11T22:34:23.910 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:34:23.910 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 100 0 100 0 0 48300 0 0 16 16 active+undersized+degraded 2020-03-11T22:33:47.576341+0000 29'100 39:119 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:33:04.364850+0000 0'0 2020-03-11T22:33:04.364850+0000 0 2020-03-11T22:34:23.911 INFO:tasks.workunit.client.0.smithi059.stdout: 2020-03-11T22:34:23.911 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:34:23.912 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2020-03-11T22:34:23.920 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:161: TEST_divergent: for i in $non_divergent 2020-03-11T22:34:23.921 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: get_asok_path osd.0 2020-03-11T22:34:23.921 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-11T22:34:23.921 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-11T22:34:23.922 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:34:23.922 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:34:23.922 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:34:23.922 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17931/ceph-osd.0.asok 2020-03-11T22:34:23.922 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: CEPH_ARGS= 2020-03-11T22:34:23.922 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.17931/ceph-osd.0.asok set_recovery_delay 100000 2020-03-11T22:34:24.018 INFO:tasks.workunit.client.0.smithi059.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:34:24.025 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:161: TEST_divergent: for i in $non_divergent 2020-03-11T22:34:24.026 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: get_asok_path osd.2 2020-03-11T22:34:24.026 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-03-11T22:34:24.026 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-03-11T22:34:24.026 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:34:24.027 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:34:24.027 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:34:24.027 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17931/ceph-osd.2.asok 2020-03-11T22:34:24.027 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: CEPH_ARGS= 2020-03-11T22:34:24.027 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.17931/ceph-osd.2.asok set_recovery_delay 100000 2020-03-11T22:34:24.123 INFO:tasks.workunit.client.0.smithi059.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:34:24.130 INFO:tasks.workunit.client.0.smithi059.stdout:revive divergent 1 2020-03-11T22:34:24.130 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:167: TEST_divergent: echo 'revive divergent 1' 2020-03-11T22:34:24.130 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:168: TEST_divergent: ceph pg dump pgs 2020-03-11T22:34:24.473 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:34:24.474 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 100 0 100 0 0 48300 0 0 16 16 active+undersized+degraded 2020-03-11T22:33:47.576341+0000 29'100 39:119 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:33:04.364850+0000 0'0 2020-03-11T22:33:04.364850+0000 0 2020-03-11T22:34:24.474 INFO:tasks.workunit.client.0.smithi059.stdout: 2020-03-11T22:34:24.474 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:34:24.475 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2020-03-11T22:34:24.483 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:169: TEST_divergent: ceph osd set noup 2020-03-11T22:34:25.189 INFO:tasks.workunit.client.0.smithi059.stderr:noup is set 2020-03-11T22:34:25.200 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:170: TEST_divergent: activate_osd td/divergent-priors 1 2020-03-11T22:34:25.200 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:34:25.200 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:34:25.201 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-11T22:34:25.201 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:34:25.201 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:34:25.201 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=cb7f2e42-1934-4145-a3c3-29bdd93f8adb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:34:25.201 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:34:25.201 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:34:25.201 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:34:25.202 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:34:25.202 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:34:25.202 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:34:25.202 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:34:25.202 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:34:25.202 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:34:25.202 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:34:25.203 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:34:25.203 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:34:25.203 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:34:25.203 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:34:25.203 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17931/$cluster-$name.asok' 2020-03-11T22:34:25.203 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' 2020-03-11T22:34:25.203 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:34:25.204 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:34:25.204 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:34:25.204 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:34:25.204 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:34:25.204 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:34:25.204 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:34:25.205 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:34:25.205 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:34:25.205 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-11T22:34:25.205 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=cb7f2e42-1934-4145-a3c3-29bdd93f8adb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:34:25.205 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-03-11T22:34:25.205 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.1 2020-03-11T22:34:25.206 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-11T22:34:25.206 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:34:25.206 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:34:25.206 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:34:25.228 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:34:25.225+0000 7f49ebe63ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:34:25.230 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:34:25.227+0000 7f49ebe63ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:34:25.232 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:34:25.229+0000 7f49ebe63ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:34:25.563 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: return 0 2020-03-11T22:34:25.563 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:171: TEST_divergent: sleep 5 2020-03-11T22:34:25.748 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:34:25.745+0000 7f49ebe63ec0 -1 Falling back to public interface 2020-03-11T22:34:26.606 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:34:26.603+0000 7f49ebe63ec0 -1 osd.1 29 log_to_monitors {default=true} 2020-03-11T22:34:30.565 INFO:tasks.workunit.client.0.smithi059.stdout:delay recovery divergent 2020-03-11T22:34:30.565 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:173: TEST_divergent: echo 'delay recovery divergent' 2020-03-11T22:34:30.566 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:174: TEST_divergent: ceph pg dump pgs 2020-03-11T22:34:30.914 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:34:30.915 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2020-03-11T22:33:47.576341+0000 39'101 39:121 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:33:04.364850+0000 0'0 2020-03-11T22:33:04.364850+0000 0 2020-03-11T22:34:30.915 INFO:tasks.workunit.client.0.smithi059.stdout: 2020-03-11T22:34:30.915 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:34:30.916 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2020-03-11T22:34:30.925 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: get_asok_path osd.1 2020-03-11T22:34:30.925 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:34:30.925 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:34:30.925 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:34:30.925 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:34:30.925 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:34:30.926 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17931/ceph-osd.1.asok 2020-03-11T22:34:30.926 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: CEPH_ARGS= 2020-03-11T22:34:30.926 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.17931/ceph-osd.1.asok set_recovery_delay 100000 2020-03-11T22:34:31.023 INFO:tasks.workunit.client.0.smithi059.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:34:31.030 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:177: TEST_divergent: ceph osd unset noup 2020-03-11T22:34:31.802 INFO:tasks.workunit.client.0.smithi059.stderr:noup is unset 2020-03-11T22:34:31.811 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:179: TEST_divergent: wait_for_osd up 0 2020-03-11T22:34:31.812 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:34:31.812 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:34:31.812 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:34:31.812 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:34:31.812 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:34:31.812 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:34:31.812 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:34:31.813 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:34:31.813 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:34:32.180 INFO:tasks.workunit.client.0.smithi059.stdout:osd.0 up in weight 1 up_from 35 up_thru 38 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/23083,v1:127.0.0.1:6803/23083] [v2:127.0.0.1:6804/23083,v1:127.0.0.1:6805/23083] exists,up 3024509c-3f4d-4606-be45-2b0ba42736ad 2020-03-11T22:34:32.180 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:34:32.181 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:34:32.181 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:34:32.181 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:34:32.181 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:180: TEST_divergent: wait_for_osd up 1 2020-03-11T22:34:32.181 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:34:32.181 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:34:32.181 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:34:32.182 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:34:32.182 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:34:32.182 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:34:32.182 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:34:32.182 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:34:32.587 INFO:tasks.workunit.client.0.smithi059.stdout:osd.1 up in weight 1 up_from 43 up_thru 43 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/24737,v1:127.0.0.1:6819/24737] [v2:127.0.0.1:6820/24737,v1:127.0.0.1:6821/24737] exists,up 85494619-d02f-47a9-a790-661f9ba109cb 2020-03-11T22:34:32.587 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:34:32.587 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:34:32.587 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:34:32.587 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:34:32.587 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:181: TEST_divergent: wait_for_osd up 2 2020-03-11T22:34:32.588 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:34:32.588 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:34:32.588 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:34:32.588 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:34:32.588 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:34:32.588 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:34:32.588 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:34:32.589 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:34:32.957 INFO:tasks.workunit.client.0.smithi059.stdout:osd.2 up in weight 1 up_from 36 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/23277,v1:127.0.0.1:6811/23277] [v2:127.0.0.1:6812/23277,v1:127.0.0.1:6813/23277] exists,up 9a5ddff4-d2b7-43cd-b7c4-5b04744a990c 2020-03-11T22:34:32.957 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:34:32.957 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:34:32.957 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:34:32.957 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:183: TEST_divergent: ceph pg dump pgs 2020-03-11T22:34:33.300 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:34:33.300 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2020-03-11T22:33:47.576341+0000 39'101 39:121 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:33:04.364850+0000 0'0 2020-03-11T22:33:04.364850+0000 0 2020-03-11T22:34:33.300 INFO:tasks.workunit.client.0.smithi059.stdout: 2020-03-11T22:34:33.300 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:34:33.301 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2020-03-11T22:34:33.309 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:184: TEST_divergent: echo 'wait for peering' 2020-03-11T22:34:33.310 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:185: TEST_divergent: ceph pg dump pgs 2020-03-11T22:34:33.310 INFO:tasks.workunit.client.0.smithi059.stdout:wait for peering 2020-03-11T22:34:33.657 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:34:33.657 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2020-03-11T22:33:47.576341+0000 39'101 39:121 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:33:04.364850+0000 0'0 2020-03-11T22:33:04.364850+0000 0 2020-03-11T22:34:33.657 INFO:tasks.workunit.client.0.smithi059.stdout: 2020-03-11T22:34:33.658 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:34:33.658 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2020-03-11T22:34:33.666 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:186: TEST_divergent: rados -p test put foo /etc/fstab 2020-03-11T22:34:43.614 INFO:tasks.workunit.client.0.smithi059.stdout:killing divergent 1 2020-03-11T22:34:43.614 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:188: TEST_divergent: echo 'killing divergent 1' 2020-03-11T22:34:43.614 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:189: TEST_divergent: ceph pg dump pgs 2020-03-11T22:34:43.965 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:34:43.965 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 100 2 2 0 0 47947 0 0 17 17 active+recovering+degraded+wait 2020-03-11T22:34:32.073311+0000 39'101 45:133 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:33:04.364850+0000 0'0 2020-03-11T22:33:04.364850+0000 0 2020-03-11T22:34:43.965 INFO:tasks.workunit.client.0.smithi059.stdout: 2020-03-11T22:34:43.965 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:34:43.966 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2020-03-11T22:34:43.974 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:190: TEST_divergent: kill_daemons td/divergent-priors KILL osd.1 2020-03-11T22:34:43.975 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:34:43.975 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:34:43.975 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:34:43.976 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:34:43.976 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:34:44.080 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:34:44.080 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:192: TEST_divergent: echo 'reviving divergent 1' 2020-03-11T22:34:44.080 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:193: TEST_divergent: ceph pg dump pgs 2020-03-11T22:34:44.081 INFO:tasks.workunit.client.0.smithi059.stdout:reviving divergent 1 2020-03-11T22:34:44.423 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:34:44.423 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 100 2 2 0 0 47947 0 0 17 17 active+recovering+degraded+wait 2020-03-11T22:34:32.073311+0000 39'101 45:133 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:33:04.364850+0000 0'0 2020-03-11T22:33:04.364850+0000 0 2020-03-11T22:34:44.423 INFO:tasks.workunit.client.0.smithi059.stdout: 2020-03-11T22:34:44.423 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:34:44.424 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2020-03-11T22:34:44.436 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:194: TEST_divergent: activate_osd td/divergent-priors 1 2020-03-11T22:34:44.436 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:34:44.436 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:34:44.436 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-11T22:34:44.436 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:34:44.437 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:34:44.437 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=cb7f2e42-1934-4145-a3c3-29bdd93f8adb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:34:44.437 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:34:44.437 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:34:44.437 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:34:44.437 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:34:44.437 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:34:44.438 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:34:44.438 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:34:44.438 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:34:44.438 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:34:44.438 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:34:44.439 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:34:44.439 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:34:44.439 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:34:44.439 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:34:44.439 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17931/$cluster-$name.asok' 2020-03-11T22:34:44.439 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' 2020-03-11T22:34:44.440 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:34:44.440 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:34:44.440 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:34:44.440 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:34:44.440 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:34:44.440 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:34:44.441 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:34:44.441 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:34:44.441 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:34:44.441 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-11T22:34:44.441 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=cb7f2e42-1934-4145-a3c3-29bdd93f8adb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:34:44.441 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-03-11T22:34:44.442 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.1 2020-03-11T22:34:44.442 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-11T22:34:44.442 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:34:44.443 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:34:44.443 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:34:44.463 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:34:44.459+0000 7f36a6b40ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:34:44.465 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:34:44.461+0000 7f36a6b40ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:34:44.466 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:34:44.463+0000 7f36a6b40ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:34:44.805 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 1 2020-03-11T22:34:44.805 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:34:44.805 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:34:44.806 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:34:44.806 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:34:44.806 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:34:44.806 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:34:44.806 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:34:44.806 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:34:44.806 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:34:44.987 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:34:44.983+0000 7f36a6b40ec0 -1 Falling back to public interface 2020-03-11T22:34:45.192 INFO:tasks.workunit.client.0.smithi059.stdout:osd.1 up in weight 1 up_from 43 up_thru 43 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/24737,v1:127.0.0.1:6819/24737] [v2:127.0.0.1:6820/24737,v1:127.0.0.1:6821/24737] exists,up 85494619-d02f-47a9-a790-661f9ba109cb 2020-03-11T22:34:45.192 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:34:45.192 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:34:45.193 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:34:45.193 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:196: TEST_divergent: sleep 20 2020-03-11T22:34:45.895 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:34:45.891+0000 7f36a6b40ec0 -1 osd.1 45 log_to_monitors {default=true} 2020-03-11T22:35:05.193 INFO:tasks.workunit.client.0.smithi059.stdout:allowing recovery 2020-03-11T22:35:05.194 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:198: TEST_divergent: echo 'allowing recovery' 2020-03-11T22:35:05.194 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:199: TEST_divergent: ceph pg dump pgs 2020-03-11T22:35:05.543 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:35:05.543 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 101 0 0 0 0 48430 0 0 18 18 active+clean 2020-03-11T22:34:46.951621+0000 45'102 48:150 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:33:04.364850+0000 0'0 2020-03-11T22:33:04.364850+0000 0 2020-03-11T22:35:05.543 INFO:tasks.workunit.client.0.smithi059.stdout: 2020-03-11T22:35:05.543 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:35:05.544 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2020-03-11T22:35:05.553 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2020-03-11T22:35:05.553 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.0 debug kick_recovery_wq 0 2020-03-11T22:35:05.680 INFO:tasks.workunit.client.0.smithi059.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:35:05.687 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2020-03-11T22:35:05.688 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.1 debug kick_recovery_wq 0 2020-03-11T22:35:05.817 INFO:tasks.workunit.client.0.smithi059.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:35:05.824 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2020-03-11T22:35:05.824 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.2 debug kick_recovery_wq 0 2020-03-11T22:35:05.952 INFO:tasks.workunit.client.0.smithi059.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:35:05.960 INFO:tasks.workunit.client.0.smithi059.stdout:reading divergent objects 2020-03-11T22:35:05.960 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:206: TEST_divergent: echo 'reading divergent objects' 2020-03-11T22:35:05.960 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:207: TEST_divergent: ceph pg dump pgs 2020-03-11T22:35:06.300 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:35:06.300 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 101 0 0 0 0 48430 0 0 18 18 active+clean 2020-03-11T22:34:46.951621+0000 45'102 48:150 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:33:04.364850+0000 0'0 2020-03-11T22:33:04.364850+0000 0 2020-03-11T22:35:06.301 INFO:tasks.workunit.client.0.smithi059.stdout: 2020-03-11T22:35:06.301 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:35:06.302 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2020-03-11T22:35:06.310 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: expr 2 + 2 2020-03-11T22:35:06.311 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: seq 1 4 2020-03-11T22:35:06.312 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:35:06.312 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_1 td/divergent-priors/existing 2020-03-11T22:35:06.356 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:35:06.356 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_2 td/divergent-priors/existing 2020-03-11T22:35:06.386 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:35:06.386 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_3 td/divergent-priors/existing 2020-03-11T22:35:06.416 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:35:06.416 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_4 td/divergent-priors/existing 2020-03-11T22:35:06.444 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:212: TEST_divergent: rm -f td/divergent-priors/existing 2020-03-11T22:35:06.445 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:214: TEST_divergent: find td/divergent-priors -name '*osd*log' 2020-03-11T22:35:06.446 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:214: TEST_divergent: grep _merge_object_divergent_entries td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:35:06.449 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:34:32.005+0000 7f49ca01e700 20 _merge_object_divergent_entries: merging hoid 1:c156033e:::existing_100:head entries: 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4526.0:1 2020-03-11T22:33:27.995958+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:35:06.449 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:34:32.005+0000 7f49ca01e700 20 _merge_object_divergent_entries: keeping 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4526.0:1 2020-03-11T22:33:27.995958+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:35:06.450 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:34:32.005+0000 7f49ca01e700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:35:06.450 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:34:32.005+0000 7f49ca01e700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head prior_version: 29'100 first_divergent_update: 29'101 last_divergent_update: 29'101 2020-03-11T22:35:06.450 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:34:32.005+0000 7f49ca01e700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head has no more recent entries in log 2020-03-11T22:35:06.450 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:34:32.005+0000 7f49ca01e700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head must be rolled back or recovered, attempting to rollback 2020-03-11T22:35:06.450 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:34:32.005+0000 7f49ca01e700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head olog_can_rollback_to: 29'101 2020-03-11T22:35:06.450 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:34:32.005+0000 7f49ca01e700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot rollback 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4526.0:1 2020-03-11T22:33:27.995958+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:35:06.451 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:34:32.005+0000 7f49ca01e700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot roll back, removing and adding to missing 2020-03-11T22:35:06.451 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:216: TEST_divergent: find td/divergent-priors -name '*osd*log' 2020-03-11T22:35:06.451 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:216: TEST_divergent: grep -q '_merge_object_divergent_entries.*cannot roll back, removing and adding to missing' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:35:06.454 INFO:tasks.workunit.client.0.smithi059.stdout:success 2020-03-11T22:35:06.454 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:221: TEST_divergent: echo success 2020-03-11T22:35:06.454 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:223: TEST_divergent: delete_pool test 2020-03-11T22:35:06.454 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test 2020-03-11T22:35:06.454 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2020-03-11T22:35:06.944 INFO:tasks.workunit.client.0.smithi059.stderr:pool 'test' does not exist 2020-03-11T22:35:06.950 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:224: TEST_divergent: kill_daemons td/divergent-priors 2020-03-11T22:35:06.951 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:35:06.951 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:35:06.951 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:35:06.952 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:35:06.952 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:35:07.056 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:35:07.057 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2020-03-11T22:35:07.057 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-11T22:35:07.057 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:35:07.057 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-11T22:35:07.057 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:35:07.057 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:35:07.058 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:35:07.058 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:35:07.058 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:35:07.067 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:35:07.067 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:35:07.067 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:35:07.067 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:35:07.067 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:35:07.067 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:35:07.068 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:35:07.068 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:35:07.068 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:35:07.068 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:35:07.068 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:35:07.068 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:35:07.069 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:35:07.069 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-11T22:35:07.079 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:35:07.079 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:35:07.080 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:35:07.080 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17931 2020-03-11T22:35:07.081 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:35:07.081 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:35:07.081 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2020-03-11T22:35:07.082 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2020-03-11T22:35:07.082 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2020-03-11T22:35:07.082 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2020-03-11T22:35:07.082 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-11T22:35:07.082 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:35:07.082 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-11T22:35:07.083 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:35:07.083 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:35:07.083 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:35:07.083 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:35:07.083 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:35:07.084 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:35:07.084 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:35:07.084 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:35:07.085 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:35:07.085 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:35:07.085 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:35:07.085 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:35:07.086 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:35:07.086 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:35:07.086 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:35:07.086 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:35:07.098 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:35:07.098 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:35:07.098 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-11T22:35:07.101 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:35:07.101 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:35:07.101 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:35:07.101 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17931 2020-03-11T22:35:07.101 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:35:07.101 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:35:07.102 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2020-03-11T22:35:07.102 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-11T22:35:07.102 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:35:07.102 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:35:07.102 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.17931 2020-03-11T22:35:07.102 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-11T22:35:07.102 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-03-11T22:35:07.103 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-11T22:35:07.103 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2020-03-11T22:35:07.103 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_2 td/divergent-priors 2020-03-11T22:35:07.103 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:421: TEST_divergent_2: local dir=td/divergent-priors 2020-03-11T22:35:07.103 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:424: TEST_divergent_2: local dummyfile=/etc/fstab 2020-03-11T22:35:07.103 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:425: TEST_divergent_2: local dummyfile2=/etc/resolv.conf 2020-03-11T22:35:07.103 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:427: TEST_divergent_2: local num_osds=3 2020-03-11T22:35:07.104 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: expr 3 - 1 2020-03-11T22:35:07.105 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: seq 0 2 2020-03-11T22:35:07.105 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: local 'osds=0 2020-03-11T22:35:07.106 INFO:tasks.workunit.client.0.smithi059.stderr:1 2020-03-11T22:35:07.106 INFO:tasks.workunit.client.0.smithi059.stderr:2' 2020-03-11T22:35:07.106 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:429: TEST_divergent_2: run_mon td/divergent-priors a 2020-03-11T22:35:07.106 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2020-03-11T22:35:07.106 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-11T22:35:07.106 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-11T22:35:07.107 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-11T22:35:07.107 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2020-03-11T22:35:07.107 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2020-03-11T22:35:07.424 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-11T22:35:07.424 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:35:07.424 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:35:07.424 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:35:07.424 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:35:07.425 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:35:07.425 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17931/$cluster-$name.asok' 2020-03-11T22:35:07.425 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-11T22:35:07.608 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-11T22:35:07.609 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-11T22:35:07.609 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:35:07.609 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:35:07.610 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-11T22:35:07.610 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-11T22:35:07.610 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:35:07.611 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:35:07.611 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:35:07.611 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:35:07.611 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:35:07.612 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:35:07.612 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17931/ceph-mon.a.asok 2020-03-11T22:35:07.612 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:35:07.612 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.17931/ceph-mon.a.asok config get fsid 2020-03-11T22:35:07.703 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-11T22:35:07.703 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:35:07.703 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:35:07.703 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-11T22:35:07.704 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-11T22:35:07.704 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:35:07.704 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:35:07.704 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:35:07.704 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:35:07.704 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:35:07.704 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:35:07.705 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17931/ceph-mon.a.asok 2020-03-11T22:35:07.705 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:35:07.705 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.17931/ceph-mon.a.asok config get mon_host 2020-03-11T22:35:07.793 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:430: TEST_divergent_2: run_mgr td/divergent-priors x 2020-03-11T22:35:07.793 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/divergent-priors 2020-03-11T22:35:07.794 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-11T22:35:07.794 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-11T22:35:07.794 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-11T22:35:07.794 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/divergent-priors/x 2020-03-11T22:35:07.794 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-11T22:35:08.276 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-11T22:35:08.276 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:35:08.276 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:35:08.276 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:35:08.277 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:35:08.277 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:35:08.277 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17931/$cluster-$name.asok' 2020-03-11T22:35:08.277 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:35:08.278 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:35:08.309 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2020-03-11T22:35:08.309 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 0 2020-03-11T22:35:08.309 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:35:08.310 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:35:08.310 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-11T22:35:08.310 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:35:08.310 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/0 2020-03-11T22:35:08.310 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=cb7f2e42-1934-4145-a3c3-29bdd93f8adb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:35:08.310 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:35:08.311 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:35:08.311 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:35:08.311 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-11T22:35:08.311 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-11T22:35:08.311 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:35:08.312 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:35:08.312 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:35:08.312 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:35:08.312 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:35:08.312 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:35:08.312 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:35:08.312 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:35:08.312 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:35:08.313 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17931/$cluster-$name.asok' 2020-03-11T22:35:08.313 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' 2020-03-11T22:35:08.313 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:35:08.313 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:35:08.313 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:35:08.313 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:35:08.314 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:35:08.314 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:35:08.314 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:35:08.314 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:35:08.314 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:35:08.314 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:35:08.314 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/0 2020-03-11T22:35:08.315 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:35:08.315 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=6b6c1a2d-a55b-410f-9b5c-d5f724d8ff3d 2020-03-11T22:35:08.315 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 6b6c1a2d-a55b-410f-9b5c-d5f724d8ff3d' 2020-03-11T22:35:08.315 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:35:08.315 INFO:tasks.workunit.client.0.smithi059.stdout:add osd0 6b6c1a2d-a55b-410f-9b5c-d5f724d8ff3d 2020-03-11T22:35:08.326 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQCcZ2leigpLExAAtr1QdAD1TEPRtiTu7uLLew== 2020-03-11T22:35:08.326 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQCcZ2leigpLExAAtr1QdAD1TEPRtiTu7uLLew=="}' 2020-03-11T22:35:08.326 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 6b6c1a2d-a55b-410f-9b5c-d5f724d8ff3d -i td/divergent-priors/0/new.json 2020-03-11T22:35:08.658 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:35:08.668 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/0/new.json 2020-03-11T22:35:08.669 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=cb7f2e42-1934-4145-a3c3-29bdd93f8adb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCcZ2leigpLExAAtr1QdAD1TEPRtiTu7uLLew== --osd-uuid 6b6c1a2d-a55b-410f-9b5c-d5f724d8ff3d 2020-03-11T22:35:08.698 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:35:08.696+0000 7f652b071ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:08.700 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:35:08.697+0000 7f652b071ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:08.702 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:35:08.699+0000 7f652b071ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:08.703 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:35:08.700+0000 7f652b071ec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:35:08.703 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:35:08.700+0000 7f652b071ec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:35:08.703 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:35:08.700+0000 7f652b071ec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:35:08.773 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:35:08.770+0000 7f652b071ec0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2020-03-11T22:35:11.846 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/0/keyring 2020-03-11T22:35:11.846 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:35:11.847 INFO:tasks.workunit.client.0.smithi059.stdout:adding osd0 key to auth repository 2020-03-11T22:35:11.847 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-11T22:35:11.847 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:35:12.342 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-11T22:35:12.342 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=cb7f2e42-1934-4145-a3c3-29bdd93f8adb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:35:12.342 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:35:12.342 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:35:12.342 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:35:12.343 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.0 2020-03-11T22:35:12.369 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:35:12.367+0000 7f7644175ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:12.371 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:35:12.368+0000 7f7644175ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:12.373 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:35:12.370+0000 7f7644175ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:12.714 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-11T22:35:12.714 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:35:12.715 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:35:12.715 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:35:12.715 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:35:12.715 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:35:12.717 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:35:12.717 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:35:12.717 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:35:12.717 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:35:12.887 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:35:12.884+0000 7f7644175ec0 -1 Falling back to public interface 2020-03-11T22:35:13.095 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:35:13.654 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:35:13.651+0000 7f7644175ec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-11T22:35:14.096 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:35:14.096 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:35:14.096 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:35:14.097 INFO:tasks.workunit.client.0.smithi059.stdout:1 2020-03-11T22:35:14.097 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:35:14.097 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:35:14.474 INFO:tasks.workunit.client.0.smithi059.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/26039,v1:127.0.0.1:6803/26039] [v2:127.0.0.1:6804/26039,v1:127.0.0.1:6805/26039] exists,up 6b6c1a2d-a55b-410f-9b5c-d5f724d8ff3d 2020-03-11T22:35:14.475 INFO:tasks.workunit.client.0.smithi059.stdout:add osd1 55fb3134-fe56-4d99-9135-8d0c47f417da 2020-03-11T22:35:14.475 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:35:14.475 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:35:14.475 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:35:14.475 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2020-03-11T22:35:14.476 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 1 2020-03-11T22:35:14.476 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:35:14.476 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:35:14.476 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-11T22:35:14.476 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:35:14.476 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:35:14.477 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=cb7f2e42-1934-4145-a3c3-29bdd93f8adb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:35:14.477 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:35:14.477 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:35:14.477 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:35:14.477 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:35:14.477 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:35:14.477 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:35:14.478 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:35:14.478 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:35:14.478 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:35:14.478 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:35:14.478 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:35:14.478 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:35:14.478 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:35:14.479 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:35:14.479 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17931/$cluster-$name.asok' 2020-03-11T22:35:14.479 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' 2020-03-11T22:35:14.479 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:35:14.479 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:35:14.479 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:35:14.479 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:35:14.480 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:35:14.480 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:35:14.480 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:35:14.480 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:35:14.480 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:35:14.480 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:35:14.480 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:35:14.480 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:35:14.481 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=55fb3134-fe56-4d99-9135-8d0c47f417da 2020-03-11T22:35:14.481 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 55fb3134-fe56-4d99-9135-8d0c47f417da' 2020-03-11T22:35:14.481 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:35:14.494 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQCiZ2ledjdIHRAARpg4sMva7KQua/nRox7OAQ== 2020-03-11T22:35:14.494 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQCiZ2ledjdIHRAARpg4sMva7KQua/nRox7OAQ=="}' 2020-03-11T22:35:14.494 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 55fb3134-fe56-4d99-9135-8d0c47f417da -i td/divergent-priors/1/new.json 2020-03-11T22:35:15.030 INFO:tasks.workunit.client.0.smithi059.stdout:1 2020-03-11T22:35:15.038 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/1/new.json 2020-03-11T22:35:15.039 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=cb7f2e42-1934-4145-a3c3-29bdd93f8adb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCiZ2ledjdIHRAARpg4sMva7KQua/nRox7OAQ== --osd-uuid 55fb3134-fe56-4d99-9135-8d0c47f417da 2020-03-11T22:35:15.069 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:35:15.066+0000 7f858e1deec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:15.071 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:35:15.068+0000 7f858e1deec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:15.072 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:35:15.069+0000 7f858e1deec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:15.073 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:35:15.070+0000 7f858e1deec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:35:15.073 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:35:15.070+0000 7f858e1deec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:35:15.073 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:35:15.070+0000 7f858e1deec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:35:15.177 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:35:15.175+0000 7f858e1deec0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2020-03-11T22:35:18.143 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/1/keyring 2020-03-11T22:35:18.144 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:35:18.145 INFO:tasks.workunit.client.0.smithi059.stdout:adding osd1 key to auth repository 2020-03-11T22:35:18.145 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-03-11T22:35:18.145 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:35:18.629 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.1 2020-03-11T22:35:18.630 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-03-11T22:35:18.630 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=cb7f2e42-1934-4145-a3c3-29bdd93f8adb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:35:18.630 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:35:18.631 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:35:18.631 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:35:18.657 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:35:18.654+0000 7f55f6748ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:18.659 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:35:18.656+0000 7f55f6748ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:18.661 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:35:18.658+0000 7f55f6748ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:19.000 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-03-11T22:35:19.000 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:35:19.000 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:35:19.000 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:35:19.001 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:35:19.001 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:35:19.001 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:35:19.001 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:35:19.001 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:35:19.001 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:35:19.178 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:35:19.175+0000 7f55f6748ec0 -1 Falling back to public interface 2020-03-11T22:35:19.366 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:35:19.968 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:35:19.965+0000 7f55f6748ec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-11T22:35:20.368 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:35:20.369 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:35:20.369 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:35:20.369 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:35:20.370 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:35:20.370 INFO:tasks.workunit.client.0.smithi059.stdout:1 2020-03-11T22:35:20.749 INFO:tasks.workunit.client.0.smithi059.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/26394,v1:127.0.0.1:6811/26394] [v2:127.0.0.1:6812/26394,v1:127.0.0.1:6813/26394] exists,up 55fb3134-fe56-4d99-9135-8d0c47f417da 2020-03-11T22:35:20.749 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:35:20.749 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:35:20.749 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:35:20.749 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2020-03-11T22:35:20.750 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 2 2020-03-11T22:35:20.750 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:35:20.750 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:35:20.750 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-03-11T22:35:20.750 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:35:20.750 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/2 2020-03-11T22:35:20.750 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=cb7f2e42-1934-4145-a3c3-29bdd93f8adb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:35:20.751 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:35:20.751 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:35:20.751 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:35:20.751 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-11T22:35:20.751 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-11T22:35:20.751 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:35:20.751 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:35:20.752 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:35:20.752 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:35:20.752 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:35:20.752 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:35:20.752 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:35:20.752 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:35:20.752 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:35:20.752 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17931/$cluster-$name.asok' 2020-03-11T22:35:20.753 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' 2020-03-11T22:35:20.753 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:35:20.753 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:35:20.753 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:35:20.753 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:35:20.753 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:35:20.753 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:35:20.754 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:35:20.754 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:35:20.754 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:35:20.754 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:35:20.754 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/2 2020-03-11T22:35:20.754 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:35:20.755 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=3aa1b594-c731-4355-b2e6-05a8c3d7bb4b 2020-03-11T22:35:20.755 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 3aa1b594-c731-4355-b2e6-05a8c3d7bb4b' 2020-03-11T22:35:20.755 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:35:20.756 INFO:tasks.workunit.client.0.smithi059.stdout:add osd2 3aa1b594-c731-4355-b2e6-05a8c3d7bb4b 2020-03-11T22:35:20.768 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQCoZ2leTWajLRAAHPUdfZTreamWhsgYxutLnw== 2020-03-11T22:35:20.768 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQCoZ2leTWajLRAAHPUdfZTreamWhsgYxutLnw=="}' 2020-03-11T22:35:20.768 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 3aa1b594-c731-4355-b2e6-05a8c3d7bb4b -i td/divergent-priors/2/new.json 2020-03-11T22:35:21.312 INFO:tasks.workunit.client.0.smithi059.stdout:2 2020-03-11T22:35:21.322 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/2/new.json 2020-03-11T22:35:21.323 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=cb7f2e42-1934-4145-a3c3-29bdd93f8adb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCoZ2leTWajLRAAHPUdfZTreamWhsgYxutLnw== --osd-uuid 3aa1b594-c731-4355-b2e6-05a8c3d7bb4b 2020-03-11T22:35:21.353 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:35:21.350+0000 7f87965e4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:21.366 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:35:21.364+0000 7f87965e4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:21.368 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:35:21.365+0000 7f87965e4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:21.369 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:35:21.366+0000 7f87965e4ec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:35:21.369 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:35:21.366+0000 7f87965e4ec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:35:21.369 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:35:21.366+0000 7f87965e4ec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:35:21.450 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:35:21.447+0000 7f87965e4ec0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2020-03-11T22:35:24.554 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/2/keyring 2020-03-11T22:35:24.555 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:35:24.556 INFO:tasks.workunit.client.0.smithi059.stdout:adding osd2 key to auth repository 2020-03-11T22:35:24.556 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-03-11T22:35:24.556 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:35:25.040 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-03-11T22:35:25.041 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=cb7f2e42-1934-4145-a3c3-29bdd93f8adb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:35:25.041 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:35:25.041 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:35:25.041 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:35:25.042 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.2 2020-03-11T22:35:25.068 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:35:25.065+0000 7ff8f0f4dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:25.092 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:35:25.090+0000 7ff8f0f4dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:25.093 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:35:25.090+0000 7ff8f0f4dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:25.405 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-03-11T22:35:25.405 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:35:25.405 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:35:25.405 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:35:25.405 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:35:25.406 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:35:25.406 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:35:25.406 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:35:25.406 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:35:25.406 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:35:25.609 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:35:25.606+0000 7ff8f0f4dec0 -1 Falling back to public interface 2020-03-11T22:35:25.774 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:35:26.399 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:35:26.397+0000 7ff8f0f4dec0 -1 osd.2 0 log_to_monitors {default=true} 2020-03-11T22:35:26.776 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:35:26.776 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:35:26.776 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:35:26.776 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:35:26.777 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:35:26.777 INFO:tasks.workunit.client.0.smithi059.stdout:1 2020-03-11T22:35:27.155 INFO:tasks.workunit.client.0.smithi059.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/26749,v1:127.0.0.1:6819/26749] [v2:127.0.0.1:6820/26749,v1:127.0.0.1:6821/26749] exists,up 3aa1b594-c731-4355-b2e6-05a8c3d7bb4b 2020-03-11T22:35:27.155 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:35:27.155 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:35:27.156 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:35:27.156 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:436: TEST_divergent_2: ceph osd set noout 2020-03-11T22:35:27.976 INFO:tasks.workunit.client.0.smithi059.stderr:noout is set 2020-03-11T22:35:27.984 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:437: TEST_divergent_2: ceph osd set noin 2020-03-11T22:35:28.627 INFO:tasks.workunit.client.0.smithi059.stderr:noin is set 2020-03-11T22:35:28.635 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:438: TEST_divergent_2: ceph osd set nodown 2020-03-11T22:35:29.387 INFO:tasks.workunit.client.0.smithi059.stderr:nodown is set 2020-03-11T22:35:29.395 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:439: TEST_divergent_2: create_pool test 1 1 2020-03-11T22:35:29.395 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test 1 1 2020-03-11T22:35:29.946 INFO:tasks.workunit.client.0.smithi059.stderr:pool 'test' already exists 2020-03-11T22:35:29.954 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:35:30.955 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:440: TEST_divergent_2: ceph osd pool set test size 3 2020-03-11T22:35:31.574 INFO:tasks.workunit.client.0.smithi059.stderr:set pool 1 size to 3 2020-03-11T22:35:31.582 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:441: TEST_divergent_2: ceph osd pool set test min_size 2 2020-03-11T22:35:32.309 INFO:tasks.workunit.client.0.smithi059.stderr:set pool 1 min_size to 2 2020-03-11T22:35:32.318 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:443: TEST_divergent_2: flush_pg_stats 2020-03-11T22:35:32.318 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:35:32.318 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:35:32.685 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:35:32.685 INFO:tasks.workunit.client.0.smithi059.stderr:1 2020-03-11T22:35:32.685 INFO:tasks.workunit.client.0.smithi059.stderr:2' 2020-03-11T22:35:32.685 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:35:32.686 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:35:32.686 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:35:32.822 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836486 2020-03-11T22:35:32.823 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486' 2020-03-11T22:35:32.823 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:35:32.823 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:35:32.960 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672965 2020-03-11T22:35:32.960 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486 1-42949672965' 2020-03-11T22:35:32.960 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:35:32.960 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:35:33.095 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509444 2020-03-11T22:35:33.095 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486 1-42949672965 2-64424509444' 2020-03-11T22:35:33.095 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:35:33.095 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836486 2020-03-11T22:35:33.096 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:35:33.096 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:35:33.097 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836486 2020-03-11T22:35:33.097 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:35:33.098 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836486 2020-03-11T22:35:33.098 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2020-03-11T22:35:33.098 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:35:33.098 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.0 seq 21474836486 2020-03-11T22:35:33.463 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836484 -lt 21474836486 2020-03-11T22:35:33.463 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:35:34.464 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:35:34.465 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:35:34.839 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836486 -lt 21474836486 2020-03-11T22:35:34.839 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:35:34.839 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672965 2020-03-11T22:35:34.839 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:35:34.840 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:35:34.840 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672965 2020-03-11T22:35:34.840 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:35:34.841 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672965 2020-03-11T22:35:34.841 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2020-03-11T22:35:34.842 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:35:34.842 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.1 seq 42949672965 2020-03-11T22:35:35.231 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672965 -lt 42949672965 2020-03-11T22:35:35.231 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:35:35.232 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509444 2020-03-11T22:35:35.232 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:35:35.232 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:35:35.233 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509444 2020-03-11T22:35:35.233 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:35:35.234 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509444 2020-03-11T22:35:35.234 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2020-03-11T22:35:35.234 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:35:35.235 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.2 seq 64424509444 2020-03-11T22:35:35.599 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509444 -lt 64424509444 2020-03-11T22:35:35.599 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:444: TEST_divergent_2: wait_for_clean 2020-03-11T22:35:35.599 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:35:35.599 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:35:35.599 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:35:35.600 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:35:35.600 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:35:35.600 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:35:35.600 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:35:35.600 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:35:35.600 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:35:35.600 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:35:35.693 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:35:35.694 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:35:35.694 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:35:35.694 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:35:35.694 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:35:36.055 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:35:36.056 INFO:tasks.workunit.client.0.smithi059.stderr:1 2020-03-11T22:35:36.056 INFO:tasks.workunit.client.0.smithi059.stderr:2' 2020-03-11T22:35:36.056 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:35:36.056 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:35:36.056 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:35:36.196 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836489 2020-03-11T22:35:36.196 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836489' 2020-03-11T22:35:36.196 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:35:36.196 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:35:36.334 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672967 2020-03-11T22:35:36.334 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836489 1-42949672967' 2020-03-11T22:35:36.335 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:35:36.335 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:35:36.465 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509446 2020-03-11T22:35:36.466 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836489 1-42949672967 2-64424509446' 2020-03-11T22:35:36.466 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:35:36.466 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836489 2020-03-11T22:35:36.466 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:35:36.468 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:35:36.468 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836489 2020-03-11T22:35:36.468 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:35:36.469 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836489 2020-03-11T22:35:36.469 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2020-03-11T22:35:36.469 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:35:36.469 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.0 seq 21474836489 2020-03-11T22:35:36.842 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836487 -lt 21474836489 2020-03-11T22:35:36.843 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:35:37.832 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:35:37.833 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:35:38.207 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836489 -lt 21474836489 2020-03-11T22:35:38.207 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:35:38.208 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672967 2020-03-11T22:35:38.208 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:35:38.209 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:35:38.209 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672967 2020-03-11T22:35:38.209 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:35:38.210 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672967 2020-03-11T22:35:38.210 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2020-03-11T22:35:38.210 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:35:38.211 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.1 seq 42949672967 2020-03-11T22:35:38.612 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672968 -lt 42949672967 2020-03-11T22:35:38.612 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:35:38.612 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509446 2020-03-11T22:35:38.612 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:35:38.613 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:35:38.613 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509446 2020-03-11T22:35:38.614 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:35:38.614 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509446 2020-03-11T22:35:38.615 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2020-03-11T22:35:38.615 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:35:38.615 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.2 seq 64424509446 2020-03-11T22:35:38.980 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509447 -lt 64424509446 2020-03-11T22:35:38.981 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:35:38.981 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:35:38.981 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:35:39.452 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:35:39.452 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:35:39.452 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:35:39.452 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:35:39.453 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:35:39.453 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:35:39.453 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:35:39.453 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:35:39.804 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:35:39.805 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:35:39.805 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:35:39.805 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:35:40.273 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:35:40.273 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:35:40.273 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:35:40.274 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: ceph pg dump pgs --format=json 2020-03-11T22:35:40.274 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: jq '.pg_stats[0].up_primary' 2020-03-11T22:35:40.619 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2020-03-11T22:35:40.627 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: local divergent=1 2020-03-11T22:35:40.627 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:448: TEST_divergent_2: echo 'primary and soon to be divergent is 1' 2020-03-11T22:35:40.627 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:449: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:35:40.628 INFO:tasks.workunit.client.0.smithi059.stdout:primary and soon to be divergent is 1 2020-03-11T22:35:40.967 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:35:40.967 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:35:30.651600+0000 0'0 24:11 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:35:29.715580+0000 0'0 2020-03-11T22:35:29.715580+0000 0 2020-03-11T22:35:40.967 INFO:tasks.workunit.client.0.smithi059.stdout: 2020-03-11T22:35:40.967 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:35:40.968 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2020-03-11T22:35:40.977 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:450: TEST_divergent_2: local non_divergent= 2020-03-11T22:35:40.978 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2020-03-11T22:35:40.978 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 0 = 1 ']' 2020-03-11T22:35:40.978 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:456: TEST_divergent_2: non_divergent=' 0' 2020-03-11T22:35:40.978 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2020-03-11T22:35:40.978 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 1 = 1 ']' 2020-03-11T22:35:40.978 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:454: TEST_divergent_2: continue 2020-03-11T22:35:40.979 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2020-03-11T22:35:40.979 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 2 = 1 ']' 2020-03-11T22:35:40.979 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:456: TEST_divergent_2: non_divergent=' 0 2' 2020-03-11T22:35:40.979 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:459: TEST_divergent_2: echo 'writing initial objects' 2020-03-11T22:35:40.979 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: seq 1 100 2020-03-11T22:35:40.979 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:40.979 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_1 /etc/fstab 2020-03-11T22:35:40.980 INFO:tasks.workunit.client.0.smithi059.stdout:writing initial objects 2020-03-11T22:35:41.064 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:41.065 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_2 /etc/fstab 2020-03-11T22:35:41.112 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:41.112 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_3 /etc/fstab 2020-03-11T22:35:41.153 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:41.154 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_4 /etc/fstab 2020-03-11T22:35:41.195 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:41.195 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_5 /etc/fstab 2020-03-11T22:35:41.253 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:41.254 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_6 /etc/fstab 2020-03-11T22:35:41.295 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:41.295 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_7 /etc/fstab 2020-03-11T22:35:41.337 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:41.337 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_8 /etc/fstab 2020-03-11T22:35:41.378 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:41.379 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_9 /etc/fstab 2020-03-11T22:35:41.465 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:41.465 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_10 /etc/fstab 2020-03-11T22:35:41.529 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:41.529 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_11 /etc/fstab 2020-03-11T22:35:41.615 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:41.616 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_12 /etc/fstab 2020-03-11T22:35:41.662 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:41.662 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_13 /etc/fstab 2020-03-11T22:35:41.749 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:41.749 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_14 /etc/fstab 2020-03-11T22:35:41.803 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:41.804 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_15 /etc/fstab 2020-03-11T22:35:41.845 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:41.846 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_16 /etc/fstab 2020-03-11T22:35:41.903 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:41.904 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_17 /etc/fstab 2020-03-11T22:35:41.979 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:41.979 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_18 /etc/fstab 2020-03-11T22:35:42.020 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:42.020 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_19 /etc/fstab 2020-03-11T22:35:42.087 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:42.088 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_20 /etc/fstab 2020-03-11T22:35:42.129 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:42.129 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_21 /etc/fstab 2020-03-11T22:35:42.208 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:42.208 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_22 /etc/fstab 2020-03-11T22:35:42.254 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:42.254 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_23 /etc/fstab 2020-03-11T22:35:42.367 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:42.367 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_24 /etc/fstab 2020-03-11T22:35:42.420 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:42.421 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_25 /etc/fstab 2020-03-11T22:35:42.500 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:42.501 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_26 /etc/fstab 2020-03-11T22:35:42.545 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:42.546 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_27 /etc/fstab 2020-03-11T22:35:42.587 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:42.587 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_28 /etc/fstab 2020-03-11T22:35:42.654 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:42.654 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_29 /etc/fstab 2020-03-11T22:35:42.696 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:42.696 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_30 /etc/fstab 2020-03-11T22:35:42.737 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:42.737 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_31 /etc/fstab 2020-03-11T22:35:42.779 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:42.779 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_32 /etc/fstab 2020-03-11T22:35:42.931 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:42.931 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_33 /etc/fstab 2020-03-11T22:35:43.026 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:43.026 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_34 /etc/fstab 2020-03-11T22:35:43.127 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:43.127 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_35 /etc/fstab 2020-03-11T22:35:43.171 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:43.171 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_36 /etc/fstab 2020-03-11T22:35:43.221 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:43.221 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_37 /etc/fstab 2020-03-11T22:35:43.302 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:43.303 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_38 /etc/fstab 2020-03-11T22:35:43.346 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:43.346 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_39 /etc/fstab 2020-03-11T22:35:43.387 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:43.388 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_40 /etc/fstab 2020-03-11T22:35:43.429 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:43.429 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_41 /etc/fstab 2020-03-11T22:35:43.504 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:43.504 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_42 /etc/fstab 2020-03-11T22:35:43.554 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:43.555 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_43 /etc/fstab 2020-03-11T22:35:43.596 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:43.596 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_44 /etc/fstab 2020-03-11T22:35:43.677 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:43.678 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_45 /etc/fstab 2020-03-11T22:35:43.721 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:43.721 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_46 /etc/fstab 2020-03-11T22:35:43.853 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:43.853 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_47 /etc/fstab 2020-03-11T22:35:43.904 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:43.905 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_48 /etc/fstab 2020-03-11T22:35:43.987 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:43.987 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_49 /etc/fstab 2020-03-11T22:35:44.120 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:44.120 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_50 /etc/fstab 2020-03-11T22:35:44.171 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:44.171 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_51 /etc/fstab 2020-03-11T22:35:44.212 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:44.213 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_52 /etc/fstab 2020-03-11T22:35:44.254 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:44.255 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_53 /etc/fstab 2020-03-11T22:35:44.296 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:44.296 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_54 /etc/fstab 2020-03-11T22:35:44.346 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:44.347 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_55 /etc/fstab 2020-03-11T22:35:44.388 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:44.388 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_56 /etc/fstab 2020-03-11T22:35:44.512 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:44.512 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_57 /etc/fstab 2020-03-11T22:35:44.563 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:44.563 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_58 /etc/fstab 2020-03-11T22:35:44.671 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:44.671 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_59 /etc/fstab 2020-03-11T22:35:44.721 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:44.721 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_60 /etc/fstab 2020-03-11T22:35:44.804 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:44.804 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_61 /etc/fstab 2020-03-11T22:35:44.855 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:44.855 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_62 /etc/fstab 2020-03-11T22:35:44.913 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:44.913 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_63 /etc/fstab 2020-03-11T22:35:44.955 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:44.955 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_64 /etc/fstab 2020-03-11T22:35:44.996 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:44.996 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_65 /etc/fstab 2020-03-11T22:35:45.038 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:45.038 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_66 /etc/fstab 2020-03-11T22:35:45.080 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:45.080 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_67 /etc/fstab 2020-03-11T22:35:45.121 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:45.121 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_68 /etc/fstab 2020-03-11T22:35:45.230 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:45.230 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_69 /etc/fstab 2020-03-11T22:35:45.322 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:45.322 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_70 /etc/fstab 2020-03-11T22:35:45.380 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:45.380 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_71 /etc/fstab 2020-03-11T22:35:45.430 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:45.430 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_72 /etc/fstab 2020-03-11T22:35:45.522 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:45.522 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_73 /etc/fstab 2020-03-11T22:35:45.578 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:45.578 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_74 /etc/fstab 2020-03-11T22:35:45.613 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:45.613 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_75 /etc/fstab 2020-03-11T22:35:45.655 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:45.655 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_76 /etc/fstab 2020-03-11T22:35:45.696 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:45.696 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_77 /etc/fstab 2020-03-11T22:35:45.763 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:45.763 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_78 /etc/fstab 2020-03-11T22:35:45.805 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:45.805 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_79 /etc/fstab 2020-03-11T22:35:45.931 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:45.931 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_80 /etc/fstab 2020-03-11T22:35:45.980 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:45.980 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_81 /etc/fstab 2020-03-11T22:35:46.090 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:46.090 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_82 /etc/fstab 2020-03-11T22:35:46.138 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:46.138 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_83 /etc/fstab 2020-03-11T22:35:46.223 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:46.224 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_84 /etc/fstab 2020-03-11T22:35:46.272 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:46.272 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_85 /etc/fstab 2020-03-11T22:35:46.330 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:46.330 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_86 /etc/fstab 2020-03-11T22:35:46.372 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:46.372 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_87 /etc/fstab 2020-03-11T22:35:46.413 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:46.414 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_88 /etc/fstab 2020-03-11T22:35:46.455 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:46.455 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_89 /etc/fstab 2020-03-11T22:35:46.497 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:46.497 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_90 /etc/fstab 2020-03-11T22:35:46.539 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:46.539 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_91 /etc/fstab 2020-03-11T22:35:46.648 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:46.649 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_92 /etc/fstab 2020-03-11T22:35:46.774 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:46.774 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_93 /etc/fstab 2020-03-11T22:35:46.841 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:46.841 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_94 /etc/fstab 2020-03-11T22:35:46.897 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:46.897 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_95 /etc/fstab 2020-03-11T22:35:46.983 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:46.983 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_96 /etc/fstab 2020-03-11T22:35:47.030 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:47.030 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_97 /etc/fstab 2020-03-11T22:35:47.108 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:47.108 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_98 /etc/fstab 2020-03-11T22:35:47.180 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:47.181 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_99 /etc/fstab 2020-03-11T22:35:47.222 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:35:47.222 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_100 /etc/fstab 2020-03-11T22:35:47.264 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:466: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:35:47.264 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:466: TEST_divergent_2: wait_for_clean 2020-03-11T22:35:47.264 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:35:47.264 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:35:47.265 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:35:47.265 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:35:47.265 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:35:47.266 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:35:47.266 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:35:47.266 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:35:47.266 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:35:47.266 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:35:47.323 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:35:47.323 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:35:47.324 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:35:47.324 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:35:47.324 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:35:47.690 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:35:47.690 INFO:tasks.workunit.client.0.smithi059.stderr:1 2020-03-11T22:35:47.691 INFO:tasks.workunit.client.0.smithi059.stderr:2' 2020-03-11T22:35:47.691 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:35:47.691 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:35:47.691 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:35:47.827 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836493 2020-03-11T22:35:47.827 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493' 2020-03-11T22:35:47.827 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:35:47.827 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:35:48.036 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672972 2020-03-11T22:35:48.036 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-42949672972' 2020-03-11T22:35:48.037 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:35:48.037 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:35:48.173 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509451 2020-03-11T22:35:48.173 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-42949672972 2-64424509451' 2020-03-11T22:35:48.173 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:35:48.174 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836493 2020-03-11T22:35:48.174 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:35:48.174 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:35:48.175 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836493 2020-03-11T22:35:48.175 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:35:48.176 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836493 2020-03-11T22:35:48.176 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2020-03-11T22:35:48.176 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:35:48.176 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.0 seq 21474836493 2020-03-11T22:35:48.576 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836491 -lt 21474836493 2020-03-11T22:35:48.577 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:35:49.577 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:35:49.578 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:35:49.971 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836493 -lt 21474836493 2020-03-11T22:35:49.972 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:35:49.972 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672972 2020-03-11T22:35:49.972 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:35:49.973 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:35:49.973 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672972 2020-03-11T22:35:49.973 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:35:49.974 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672972 2020-03-11T22:35:49.974 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2020-03-11T22:35:49.975 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:35:49.975 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.1 seq 42949672972 2020-03-11T22:35:50.338 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672972 -lt 42949672972 2020-03-11T22:35:50.338 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:35:50.339 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509451 2020-03-11T22:35:50.339 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:35:50.340 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:35:50.340 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509451 2020-03-11T22:35:50.340 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:35:50.341 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509451 2020-03-11T22:35:50.341 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2020-03-11T22:35:50.342 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:35:50.342 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.2 seq 64424509451 2020-03-11T22:35:50.705 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509451 -lt 64424509451 2020-03-11T22:35:50.706 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:35:50.706 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:35:50.706 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:35:51.179 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:35:51.179 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:35:51.179 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:35:51.180 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:35:51.180 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:35:51.180 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:35:51.180 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:35:51.180 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:35:51.533 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:35:51.533 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:35:51.534 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:35:51.534 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:35:52.026 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:35:52.026 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:35:52.027 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:35:52.027 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:468: TEST_divergent_2: get_pg test existing_1 2020-03-11T22:35:52.027 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1030: get_pg: local poolname=test 2020-03-11T22:35:52.028 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local objectname=existing_1 2020-03-11T22:35:52.028 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: ceph --format json osd map test existing_1 2020-03-11T22:35:52.028 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: jq -r .pgid 2020-03-11T22:35:52.397 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:468: TEST_divergent_2: local pgid=1.0 2020-03-11T22:35:52.397 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:471: TEST_divergent_2: echo 'blackholing osds 0 2' 2020-03-11T22:35:52.397 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:472: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:35:52.398 INFO:tasks.workunit.client.0.smithi059.stdout:blackholing osds 0 2 2020-03-11T22:35:52.766 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:35:52.766 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 100 0 0 0 0 48300 0 0 16 16 active+clean 2020-03-11T22:35:30.651600+0000 28'100 28:111 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:35:29.715580+0000 0'0 2020-03-11T22:35:29.715580+0000 0 2020-03-11T22:35:52.766 INFO:tasks.workunit.client.0.smithi059.stdout: 2020-03-11T22:35:52.766 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:35:52.767 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2020-03-11T22:35:52.775 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:473: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:35:52.776 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: get_asok_path osd.0 2020-03-11T22:35:52.777 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-11T22:35:52.777 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-11T22:35:52.777 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:35:52.777 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:35:52.777 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:35:52.777 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17931/ceph-osd.0.asok 2020-03-11T22:35:52.778 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: CEPH_ARGS= 2020-03-11T22:35:52.778 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.17931/ceph-osd.0.asok config set objectstore_blackhole 1 2020-03-11T22:35:52.868 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2020-03-11T22:35:52.868 INFO:tasks.workunit.client.0.smithi059.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-03-11T22:35:52.868 INFO:tasks.workunit.client.0.smithi059.stdout:} 2020-03-11T22:35:52.875 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:473: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:35:52.876 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: get_asok_path osd.2 2020-03-11T22:35:52.876 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-03-11T22:35:52.877 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-03-11T22:35:52.877 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:35:52.877 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:35:52.877 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:35:52.877 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17931/ceph-osd.2.asok 2020-03-11T22:35:52.877 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: CEPH_ARGS= 2020-03-11T22:35:52.878 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.17931/ceph-osd.2.asok config set objectstore_blackhole 1 2020-03-11T22:35:52.967 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2020-03-11T22:35:52.967 INFO:tasks.workunit.client.0.smithi059.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-03-11T22:35:52.968 INFO:tasks.workunit.client.0.smithi059.stdout:} 2020-03-11T22:35:52.975 INFO:tasks.workunit.client.0.smithi059.stdout:create new divergent objects 2020-03-11T22:35:52.975 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:479: TEST_divergent_2: echo 'create new divergent objects' 2020-03-11T22:35:52.975 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: seq 1 2 2020-03-11T22:35:52.976 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2020-03-11T22:35:52.976 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2020-03-11T22:35:52.977 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:482: TEST_divergent_2: rados -p test create newobject_1 2020-03-11T22:35:52.977 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:485: TEST_divergent_2: echo 'writing divergent objects' 2020-03-11T22:35:52.977 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:482: TEST_divergent_2: rados -p test create newobject_2 2020-03-11T22:35:52.977 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: seq 1 2 2020-03-11T22:35:52.977 INFO:tasks.workunit.client.0.smithi059.stdout:writing divergent objects 2020-03-11T22:35:52.978 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_WRITE) 2020-03-11T22:35:52.978 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_WRITE) 2020-03-11T22:35:52.978 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:488: TEST_divergent_2: rados -p test put existing_1 /etc/resolv.conf 2020-03-11T22:35:52.979 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:491: TEST_divergent_2: echo 'remove divergent objects' 2020-03-11T22:35:52.979 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:488: TEST_divergent_2: rados -p test put existing_2 /etc/resolv.conf 2020-03-11T22:35:52.979 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: seq 1 2 2020-03-11T22:35:52.979 INFO:tasks.workunit.client.0.smithi059.stdout:remove divergent objects 2020-03-11T22:35:52.979 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_REMOVE) 2020-03-11T22:35:52.980 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: expr 1 + 2 2020-03-11T22:35:52.980 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: rmi=3 2020-03-11T22:35:52.980 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_REMOVE) 2020-03-11T22:35:52.981 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:495: TEST_divergent_2: rados -p test rm existing_3 2020-03-11T22:35:52.981 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: expr 2 + 2 2020-03-11T22:35:52.982 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: rmi=4 2020-03-11T22:35:52.982 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:497: TEST_divergent_2: sleep 10 2020-03-11T22:35:52.982 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:495: TEST_divergent_2: rados -p test rm existing_4 2020-03-11T22:36:02.984 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:498: TEST_divergent_2: killall -9 rados 2020-03-11T22:36:02.988 INFO:tasks.workunit.client.0.smithi059.stderr:rados: no process found 2020-03-11T22:36:02.988 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:501: TEST_divergent_2: echo 'killing all the osds' 2020-03-11T22:36:02.989 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:502: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:36:02.989 INFO:tasks.workunit.client.0.smithi059.stdout:killing all the osds 2020-03-11T22:36:03.340 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:36:03.341 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 100 0 0 0 0 46628 0 0 16 16 active+clean 2020-03-11T22:35:30.651600+0000 28'106 28:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:35:29.715580+0000 0'0 2020-03-11T22:35:29.715580+0000 0 2020-03-11T22:36:03.341 INFO:tasks.workunit.client.0.smithi059.stdout: 2020-03-11T22:36:03.341 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:36:03.342 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2020-03-11T22:36:03.350 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:503: TEST_divergent_2: kill_daemons td/divergent-priors KILL osd 2020-03-11T22:36:03.351 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:36:03.351 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:36:03.351 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:36:03.351 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:36:03.351 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:36:03.457 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:36:03.457 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2020-03-11T22:36:03.457 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.0 2020-03-11T22:36:03.972 INFO:tasks.workunit.client.0.smithi059.stderr:osd.0 is already down. 2020-03-11T22:36:03.980 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2020-03-11T22:36:03.980 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.1 2020-03-11T22:36:04.522 INFO:tasks.workunit.client.0.smithi059.stderr:osd.1 is already down. 2020-03-11T22:36:04.530 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2020-03-11T22:36:04.530 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.2 2020-03-11T22:36:05.062 INFO:tasks.workunit.client.0.smithi059.stderr:osd.2 is already down. 2020-03-11T22:36:05.073 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:508: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:36:05.073 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:510: TEST_divergent_2: ceph osd out osd.0 2020-03-11T22:36:05.609 INFO:tasks.workunit.client.0.smithi059.stderr:osd.0 is already out. 2020-03-11T22:36:05.617 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:508: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:36:05.618 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:510: TEST_divergent_2: ceph osd out osd.2 2020-03-11T22:36:06.215 INFO:tasks.workunit.client.0.smithi059.stderr:osd.2 is already out. 2020-03-11T22:36:06.223 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:514: TEST_divergent_2: echo 'bringing up non_divergent 0 2' 2020-03-11T22:36:06.224 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:515: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:36:06.224 INFO:tasks.workunit.client.0.smithi059.stdout:bringing up non_divergent 0 2 2020-03-11T22:36:06.601 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:36:06.601 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 100 0 0 0 0 46628 0 0 16 16 stale+active+clean 2020-03-11T22:35:30.651600+0000 28'106 28:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:35:29.715580+0000 0'0 2020-03-11T22:35:29.715580+0000 0 2020-03-11T22:36:06.601 INFO:tasks.workunit.client.0.smithi059.stdout: 2020-03-11T22:36:06.601 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:36:06.602 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2020-03-11T22:36:06.611 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:516: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:36:06.612 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:518: TEST_divergent_2: activate_osd td/divergent-priors 0 2020-03-11T22:36:06.612 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:36:06.612 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:36:06.612 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=0 2020-03-11T22:36:06.612 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:36:06.612 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/0 2020-03-11T22:36:06.613 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=cb7f2e42-1934-4145-a3c3-29bdd93f8adb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:36:06.613 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:36:06.613 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:36:06.616 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:36:06.616 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-11T22:36:06.616 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-11T22:36:06.616 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:36:06.616 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:36:06.617 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:36:06.617 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:36:06.617 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:36:06.617 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:36:06.617 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:36:06.617 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:36:06.617 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:36:06.618 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17931/$cluster-$name.asok' 2020-03-11T22:36:06.618 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' 2020-03-11T22:36:06.618 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:36:06.618 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:36:06.618 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:36:06.618 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:36:06.619 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:36:06.619 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:36:06.619 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:36:06.619 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:36:06.619 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/0 2020-03-11T22:36:06.619 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.0 2020-03-11T22:36:06.620 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 0 --fsid=cb7f2e42-1934-4145-a3c3-29bdd93f8adb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:36:06.620 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/0/whoami 2020-03-11T22:36:06.620 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 0 = 0 ']' 2020-03-11T22:36:06.621 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:36:06.621 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:36:06.621 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:36:06.621 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.0 2020-03-11T22:36:06.641 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:36:06.638+0000 7fee26108ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:06.643 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:36:06.639+0000 7fee26108ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:06.645 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:36:06.641+0000 7fee26108ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:06.983 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 0 2020-03-11T22:36:06.983 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:36:06.983 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:36:06.984 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:36:06.984 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:36:06.984 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:36:06.984 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:36:06.984 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:36:06.984 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:36:06.984 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:36:07.161 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:36:07.157+0000 7fee26108ec0 -1 Falling back to public interface 2020-03-11T22:36:07.360 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:36:08.097 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:36:08.093+0000 7fee26108ec0 -1 osd.0 28 log_to_monitors {default=true} 2020-03-11T22:36:08.361 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:36:08.362 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:36:08.362 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:36:08.362 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:36:08.362 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:36:08.362 INFO:tasks.workunit.client.0.smithi059.stdout:1 2020-03-11T22:36:08.861 INFO:tasks.workunit.client.0.smithi059.stdout:osd.0 up out weight 0 up_from 34 up_thru 0 down_at 29 last_clean_interval [5,28) [v2:127.0.0.1:6802/30799,v1:127.0.0.1:6803/30799] [v2:127.0.0.1:6804/30799,v1:127.0.0.1:6805/30799] exists,up 6b6c1a2d-a55b-410f-9b5c-d5f724d8ff3d 2020-03-11T22:36:08.861 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.2 2020-03-11T22:36:08.861 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:36:08.861 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:36:08.862 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:36:08.862 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:516: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:36:08.862 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:518: TEST_divergent_2: activate_osd td/divergent-priors 2 2020-03-11T22:36:08.862 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:36:08.862 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:36:08.862 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=2 2020-03-11T22:36:08.862 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:36:08.863 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/2 2020-03-11T22:36:08.863 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=cb7f2e42-1934-4145-a3c3-29bdd93f8adb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:36:08.863 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:36:08.863 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:36:08.863 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:36:08.863 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-11T22:36:08.864 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-11T22:36:08.864 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:36:08.864 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:36:08.864 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:36:08.864 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:36:08.864 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:36:08.864 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:36:08.865 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:36:08.865 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:36:08.865 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:36:08.865 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17931/$cluster-$name.asok' 2020-03-11T22:36:08.865 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' 2020-03-11T22:36:08.866 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:36:08.866 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:36:08.866 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:36:08.866 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:36:08.866 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:36:08.866 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:36:08.866 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:36:08.867 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:36:08.867 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/2 2020-03-11T22:36:08.867 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.2 2020-03-11T22:36:08.867 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 2 --fsid=cb7f2e42-1934-4145-a3c3-29bdd93f8adb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:36:08.867 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/2/whoami 2020-03-11T22:36:08.867 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 2 = 2 ']' 2020-03-11T22:36:08.868 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:36:08.868 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:36:08.868 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:36:08.887 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:36:08.884+0000 7f641c4ddec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:08.889 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:36:08.885+0000 7f641c4ddec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:08.891 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:36:08.887+0000 7f641c4ddec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:09.229 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 2 2020-03-11T22:36:09.229 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:36:09.229 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:36:09.229 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:36:09.230 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:36:09.230 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:36:09.230 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:36:09.230 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:36:09.230 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:36:09.230 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:36:09.407 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:36:09.403+0000 7f641c4ddec0 -1 Falling back to public interface 2020-03-11T22:36:09.613 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:36:10.322 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:36:10.318+0000 7f641c4ddec0 -1 osd.2 28 log_to_monitors {default=true} 2020-03-11T22:36:10.615 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:36:10.615 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:36:10.615 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:36:10.615 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:36:10.615 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:36:10.616 INFO:tasks.workunit.client.0.smithi059.stdout:1 2020-03-11T22:36:10.990 INFO:tasks.workunit.client.0.smithi059.stdout:osd.2 up out weight 0 up_from 35 up_thru 0 down_at 31 last_clean_interval [15,28) [v2:127.0.0.1:6810/30993,v1:127.0.0.1:6811/30993] [v2:127.0.0.1:6812/30993,v1:127.0.0.1:6813/30993] exists,up 3aa1b594-c731-4355-b2e6-05a8c3d7bb4b 2020-03-11T22:36:10.990 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:36:10.990 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:36:10.990 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:36:10.990 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:520: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:36:10.991 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:522: TEST_divergent_2: ceph osd in osd.0 2020-03-11T22:36:11.553 INFO:tasks.workunit.client.0.smithi059.stderr:osd.0 is already in. 2020-03-11T22:36:11.561 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:520: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:36:11.561 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:522: TEST_divergent_2: ceph osd in osd.2 2020-03-11T22:36:12.295 INFO:tasks.workunit.client.0.smithi059.stderr:osd.2 is already in. 2020-03-11T22:36:12.305 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:525: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:36:12.305 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:525: TEST_divergent_2: wait_for_clean 2020-03-11T22:36:12.305 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:36:12.306 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:36:12.306 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:36:12.306 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:36:12.306 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:36:12.306 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:36:12.306 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:36:12.307 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:36:12.307 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:36:12.307 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:36:12.363 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:36:12.364 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:36:12.364 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:36:12.364 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:36:12.364 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:36:12.727 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:36:12.728 INFO:tasks.workunit.client.0.smithi059.stderr:1 2020-03-11T22:36:12.728 INFO:tasks.workunit.client.0.smithi059.stderr:2' 2020-03-11T22:36:12.728 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:36:12.728 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:36:12.728 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:36:12.863 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=146028888067 2020-03-11T22:36:12.864 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-146028888067' 2020-03-11T22:36:12.864 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:36:12.864 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:36:12.959 INFO:tasks.workunit.client.0.smithi059.stderr:Error ENXIO: problem getting command descriptions from osd.1 2020-03-11T22:36:12.967 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq= 2020-03-11T22:36:12.967 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-146028888067 1-' 2020-03-11T22:36:12.968 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:36:12.968 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:36:13.109 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=150323855363 2020-03-11T22:36:13.109 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-146028888067 1- 2-150323855363' 2020-03-11T22:36:13.109 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:36:13.110 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-146028888067 2020-03-11T22:36:13.110 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:36:13.110 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:36:13.111 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-146028888067 2020-03-11T22:36:13.111 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:36:13.112 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=146028888067 2020-03-11T22:36:13.112 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 146028888067' 2020-03-11T22:36:13.112 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:36:13.112 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.0 seq 146028888067 2020-03-11T22:36:13.478 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 0 -lt 146028888067 2020-03-11T22:36:13.478 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:36:14.479 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:36:14.480 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:36:14.854 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 146028888068 -lt 146028888067 2020-03-11T22:36:14.854 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:36:14.855 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1- 2020-03-11T22:36:14.855 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:36:14.855 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:36:14.856 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1- 2020-03-11T22:36:14.856 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:36:14.857 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq= 2020-03-11T22:36:14.857 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq ' 2020-03-11T22:36:14.857 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.1 seq 2020-03-11T22:36:14.857 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:36:15.231 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672975 -lt 2020-03-11T22:36:15.231 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 2079: test: 42949672975: unary operator expected 2020-03-11T22:36:15.231 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:36:15.232 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-150323855363 2020-03-11T22:36:15.232 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:36:15.232 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:36:15.233 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-150323855363 2020-03-11T22:36:15.233 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:36:15.234 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=150323855363 2020-03-11T22:36:15.234 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 150323855363' 2020-03-11T22:36:15.234 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:36:15.234 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.2 seq 150323855363 2020-03-11T22:36:15.596 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 150323855363 -lt 150323855363 2020-03-11T22:36:15.596 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:36:15.597 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:36:15.597 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:36:16.070 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:36:16.071 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:36:16.071 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:36:16.071 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:36:16.071 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:36:16.071 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:36:16.071 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:36:16.072 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:36:16.441 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:36:16.441 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:36:16.442 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:36:16.442 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:36:16.914 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:36:16.914 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:36:16.914 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:36:16.914 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:36:16.914 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:36:16.915 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:36:17.014 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:36:17.015 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:36:17.016 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:36:17.016 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:36:17.016 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:36:17.016 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:36:17.016 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:36:17.016 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:36:17.370 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:36:17.371 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:36:17.371 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:36:17.371 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:36:17.850 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:36:17.850 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:36:17.850 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:36:17.851 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:36:17.851 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:36:17.851 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:36:17.851 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:36:17.851 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:36:17.851 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:36:18.318 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:36:18.318 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:36:18.318 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 1 >= 8 )) 2020-03-11T22:36:18.319 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:36:18.319 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.2 2020-03-11T22:36:18.519 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:36:18.519 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:36:18.520 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:36:18.520 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:36:18.520 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:36:18.520 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:36:18.520 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:36:18.521 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:36:18.870 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:36:18.871 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:36:18.871 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:36:18.871 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:36:19.347 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:36:19.348 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:36:19.348 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:36:19.348 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:36:19.348 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:36:19.349 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:36:19.349 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:36:19.349 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:36:19.350 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:36:19.818 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:36:19.818 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:36:19.818 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 2 >= 8 )) 2020-03-11T22:36:19.818 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:36:19.818 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.4 2020-03-11T22:36:20.219 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:36:20.219 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:36:20.219 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:36:20.220 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:36:20.220 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:36:20.220 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:36:20.220 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:36:20.220 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:36:20.575 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:36:20.575 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:36:20.576 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:36:20.576 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:36:21.049 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:36:21.049 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:36:21.050 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:36:21.050 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:36:21.050 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:36:21.050 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:36:21.050 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:36:21.050 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:36:21.051 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:36:21.521 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:36:21.521 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:36:21.522 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 3 >= 8 )) 2020-03-11T22:36:21.522 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:36:21.522 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.8 2020-03-11T22:36:22.322 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:36:22.322 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:36:22.323 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:36:22.323 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:36:22.323 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:36:22.324 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:36:22.324 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:36:22.324 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:36:22.689 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:36:22.689 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:36:22.690 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:36:22.690 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:36:23.164 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:36:23.165 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:36:23.165 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:36:23.165 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:36:23.165 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:36:23.165 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:36:23.165 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:36:23.165 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:36:23.166 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:36:23.640 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:36:23.640 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:36:23.640 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 4 >= 8 )) 2020-03-11T22:36:23.640 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:36:23.640 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 1.6 2020-03-11T22:36:25.240 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:36:25.241 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:36:25.241 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:36:25.242 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:36:25.242 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:36:25.242 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:36:25.242 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:36:25.242 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:36:25.606 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:36:25.606 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:36:25.607 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:36:25.607 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:36:26.083 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:36:26.084 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:36:26.084 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:36:26.084 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:36:26.084 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:36:26.084 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:36:26.084 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:36:26.084 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:36:26.085 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:36:26.557 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:36:26.557 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:36:26.557 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 5 >= 8 )) 2020-03-11T22:36:26.558 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:36:26.558 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 3.2 2020-03-11T22:36:29.759 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:36:29.759 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:36:29.759 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:36:29.759 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:36:29.759 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:36:29.759 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:36:29.760 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:36:29.760 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:36:30.136 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:36:30.137 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:36:30.137 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:36:30.137 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:36:30.608 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:36:30.608 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:36:30.608 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:36:30.609 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:36:30.609 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:36:30.609 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:36:30.609 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:36:30.610 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:36:30.610 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:36:31.091 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:36:31.091 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:36:31.091 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 6 >= 8 )) 2020-03-11T22:36:31.092 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:36:31.092 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 6.4 2020-03-11T22:36:37.491 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:36:37.492 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:36:37.493 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:36:37.493 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:36:37.493 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:36:37.493 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:36:37.493 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:36:37.493 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:36:37.859 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:36:37.860 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:36:37.860 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:36:37.860 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:36:38.332 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:36:38.332 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:36:38.333 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:36:38.333 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:36:38.333 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:36:38.333 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:36:38.333 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:36:38.333 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:36:38.333 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:36:38.807 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:36:38.807 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:36:38.808 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 7 >= 8 )) 2020-03-11T22:36:38.808 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:36:38.808 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 7.3 2020-03-11T22:36:46.108 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:36:46.108 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:36:46.109 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:36:46.109 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:36:46.109 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:36:46.110 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:36:46.110 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:36:46.110 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:36:46.473 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:36:46.473 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:36:46.473 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:36:46.474 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:36:46.943 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:36:46.944 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:36:46.944 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:36:46.944 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:36:46.944 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:36:46.944 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:36:46.945 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:36:46.945 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:36:46.945 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:36:47.412 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:36:47.412 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:36:47.413 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 8 >= 8 )) 2020-03-11T22:36:47.413 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: ceph report 2020-03-11T22:36:47.875 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2020-03-11T22:36:47.875 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_fingerprint": "d95c3a98-702b-45ce-9ca1-a48c217954ec", 2020-03-11T22:36:47.876 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "15.1.0-1941-g4acc112", 2020-03-11T22:36:47.876 INFO:tasks.workunit.client.0.smithi059.stdout: "commit": "4acc1121e47039a3cdf2978ae458c10d153aa141", 2020-03-11T22:36:47.876 INFO:tasks.workunit.client.0.smithi059.stdout: "timestamp": "2020-03-11T22:36:47.872158+0000", 2020-03-11T22:36:47.876 INFO:tasks.workunit.client.0.smithi059.stdout: "tag": "", 2020-03-11T22:36:47.876 INFO:tasks.workunit.client.0.smithi059.stdout: "health": { 2020-03-11T22:36:47.877 INFO:tasks.workunit.client.0.smithi059.stdout: "status": "HEALTH_WARN", 2020-03-11T22:36:47.877 INFO:tasks.workunit.client.0.smithi059.stdout: "checks": { 2020-03-11T22:36:47.877 INFO:tasks.workunit.client.0.smithi059.stdout: "OSDMAP_FLAGS": { 2020-03-11T22:36:47.877 INFO:tasks.workunit.client.0.smithi059.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:36:47.877 INFO:tasks.workunit.client.0.smithi059.stdout: "summary": { 2020-03-11T22:36:47.878 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "nodown,noout,noin flag(s) set", 2020-03-11T22:36:47.878 INFO:tasks.workunit.client.0.smithi059.stdout: "count": 17 2020-03-11T22:36:47.878 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:47.878 INFO:tasks.workunit.client.0.smithi059.stdout: "detail": [], 2020-03-11T22:36:47.878 INFO:tasks.workunit.client.0.smithi059.stdout: "muted": false 2020-03-11T22:36:47.878 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:47.878 INFO:tasks.workunit.client.0.smithi059.stdout: "OSD_DOWN": { 2020-03-11T22:36:47.879 INFO:tasks.workunit.client.0.smithi059.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:36:47.879 INFO:tasks.workunit.client.0.smithi059.stdout: "summary": { 2020-03-11T22:36:47.879 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "1 osds down", 2020-03-11T22:36:47.879 INFO:tasks.workunit.client.0.smithi059.stdout: "count": 1 2020-03-11T22:36:47.879 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:47.879 INFO:tasks.workunit.client.0.smithi059.stdout: "detail": [ 2020-03-11T22:36:47.880 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:47.880 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "osd.1 (root=default,host=smithi059) is down" 2020-03-11T22:36:47.880 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:36:47.880 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:36:47.880 INFO:tasks.workunit.client.0.smithi059.stdout: "muted": false 2020-03-11T22:36:47.880 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:47.880 INFO:tasks.workunit.client.0.smithi059.stdout: "PG_DEGRADED": { 2020-03-11T22:36:47.881 INFO:tasks.workunit.client.0.smithi059.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:36:47.881 INFO:tasks.workunit.client.0.smithi059.stdout: "summary": { 2020-03-11T22:36:47.881 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded", 2020-03-11T22:36:47.881 INFO:tasks.workunit.client.0.smithi059.stdout: "count": 1 2020-03-11T22:36:47.881 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:47.881 INFO:tasks.workunit.client.0.smithi059.stdout: "detail": [ 2020-03-11T22:36:47.882 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:47.882 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:36:47.882 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:36:47.882 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:36:47.882 INFO:tasks.workunit.client.0.smithi059.stdout: "muted": false 2020-03-11T22:36:47.882 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:47.882 INFO:tasks.workunit.client.0.smithi059.stdout: "POOL_APP_NOT_ENABLED": { 2020-03-11T22:36:47.883 INFO:tasks.workunit.client.0.smithi059.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:36:47.883 INFO:tasks.workunit.client.0.smithi059.stdout: "summary": { 2020-03-11T22:36:47.883 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "1 pool(s) do not have an application enabled", 2020-03-11T22:36:47.883 INFO:tasks.workunit.client.0.smithi059.stdout: "count": 1 2020-03-11T22:36:47.883 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:47.883 INFO:tasks.workunit.client.0.smithi059.stdout: "detail": [ 2020-03-11T22:36:47.884 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:47.884 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "application not enabled on pool 'test'" 2020-03-11T22:36:47.884 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:47.884 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:47.884 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2020-03-11T22:36:47.884 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:36:47.885 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:36:47.885 INFO:tasks.workunit.client.0.smithi059.stdout: "muted": false 2020-03-11T22:36:47.885 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:36:47.885 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:47.885 INFO:tasks.workunit.client.0.smithi059.stdout: "mutes": [] 2020-03-11T22:36:47.886 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:47.886 INFO:tasks.workunit.client.0.smithi059.stdout: "monmap_first_committed": 1, 2020-03-11T22:36:47.886 INFO:tasks.workunit.client.0.smithi059.stdout: "monmap_last_committed": 1, 2020-03-11T22:36:47.886 INFO:tasks.workunit.client.0.smithi059.stdout: "monmap": { 2020-03-11T22:36:47.886 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": 1, 2020-03-11T22:36:47.886 INFO:tasks.workunit.client.0.smithi059.stdout: "fsid": "cb7f2e42-1934-4145-a3c3-29bdd93f8adb", 2020-03-11T22:36:47.887 INFO:tasks.workunit.client.0.smithi059.stdout: "modified": "2020-03-11T22:35:07.115245Z", 2020-03-11T22:36:47.887 INFO:tasks.workunit.client.0.smithi059.stdout: "created": "2020-03-11T22:35:07.115245Z", 2020-03-11T22:36:47.887 INFO:tasks.workunit.client.0.smithi059.stdout: "min_mon_release": 15, 2020-03-11T22:36:47.887 INFO:tasks.workunit.client.0.smithi059.stdout: "min_mon_release_name": "octopus", 2020-03-11T22:36:47.887 INFO:tasks.workunit.client.0.smithi059.stdout: "features": { 2020-03-11T22:36:47.887 INFO:tasks.workunit.client.0.smithi059.stdout: "persistent": [ 2020-03-11T22:36:47.888 INFO:tasks.workunit.client.0.smithi059.stdout: "kraken", 2020-03-11T22:36:47.888 INFO:tasks.workunit.client.0.smithi059.stdout: "luminous", 2020-03-11T22:36:47.888 INFO:tasks.workunit.client.0.smithi059.stdout: "mimic", 2020-03-11T22:36:47.888 INFO:tasks.workunit.client.0.smithi059.stdout: "osdmap-prune", 2020-03-11T22:36:47.888 INFO:tasks.workunit.client.0.smithi059.stdout: "nautilus", 2020-03-11T22:36:47.889 INFO:tasks.workunit.client.0.smithi059.stdout: "octopus" 2020-03-11T22:36:47.889 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:36:47.889 INFO:tasks.workunit.client.0.smithi059.stdout: "optional": [] 2020-03-11T22:36:47.889 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:47.889 INFO:tasks.workunit.client.0.smithi059.stdout: "mons": [ 2020-03-11T22:36:47.890 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:47.890 INFO:tasks.workunit.client.0.smithi059.stdout: "rank": 0, 2020-03-11T22:36:47.890 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "a", 2020-03-11T22:36:47.890 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addrs": { 2020-03-11T22:36:47.890 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:36:47.891 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:47.891 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:36:47.891 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:7115", 2020-03-11T22:36:47.891 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 0 2020-03-11T22:36:47.891 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:36:47.891 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:36:47.891 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:47.892 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:7115/0", 2020-03-11T22:36:47.892 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addr": "127.0.0.1:7115/0", 2020-03-11T22:36:47.892 INFO:tasks.workunit.client.0.smithi059.stdout: "priority": 0, 2020-03-11T22:36:47.892 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 0 2020-03-11T22:36:47.892 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:36:47.892 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:36:47.893 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:47.893 INFO:tasks.workunit.client.0.smithi059.stdout: "quorum": [ 2020-03-11T22:36:47.893 INFO:tasks.workunit.client.0.smithi059.stdout: 0 2020-03-11T22:36:47.893 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:36:47.893 INFO:tasks.workunit.client.0.smithi059.stdout: "osdmap": { 2020-03-11T22:36:47.893 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": 40, 2020-03-11T22:36:47.893 INFO:tasks.workunit.client.0.smithi059.stdout: "fsid": "cb7f2e42-1934-4145-a3c3-29bdd93f8adb", 2020-03-11T22:36:47.894 INFO:tasks.workunit.client.0.smithi059.stdout: "created": "2020-03-11T22:35:07.877727+0000", 2020-03-11T22:36:47.894 INFO:tasks.workunit.client.0.smithi059.stdout: "modified": "2020-03-11T22:36:12.169330+0000", 2020-03-11T22:36:47.894 INFO:tasks.workunit.client.0.smithi059.stdout: "last_up_change": "2020-03-11T22:36:10.399169+0000", 2020-03-11T22:36:47.894 INFO:tasks.workunit.client.0.smithi059.stdout: "last_in_change": "2020-03-11T22:36:11.901843+0000", 2020-03-11T22:36:47.894 INFO:tasks.workunit.client.0.smithi059.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-03-11T22:36:47.894 INFO:tasks.workunit.client.0.smithi059.stdout: "flags_num": 5800384, 2020-03-11T22:36:47.894 INFO:tasks.workunit.client.0.smithi059.stdout: "flags_set": [ 2020-03-11T22:36:47.895 INFO:tasks.workunit.client.0.smithi059.stdout: "nodown", 2020-03-11T22:36:47.895 INFO:tasks.workunit.client.0.smithi059.stdout: "noin", 2020-03-11T22:36:47.895 INFO:tasks.workunit.client.0.smithi059.stdout: "noout", 2020-03-11T22:36:47.895 INFO:tasks.workunit.client.0.smithi059.stdout: "pglog_hardlimit", 2020-03-11T22:36:47.895 INFO:tasks.workunit.client.0.smithi059.stdout: "purged_snapdirs", 2020-03-11T22:36:47.895 INFO:tasks.workunit.client.0.smithi059.stdout: "recovery_deletes", 2020-03-11T22:36:47.896 INFO:tasks.workunit.client.0.smithi059.stdout: "sortbitwise" 2020-03-11T22:36:47.896 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:36:47.896 INFO:tasks.workunit.client.0.smithi059.stdout: "crush_version": 7, 2020-03-11T22:36:47.896 INFO:tasks.workunit.client.0.smithi059.stdout: "full_ratio": 0.99000000953674316, 2020-03-11T22:36:47.896 INFO:tasks.workunit.client.0.smithi059.stdout: "backfillfull_ratio": 0.99000000953674316, 2020-03-11T22:36:47.896 INFO:tasks.workunit.client.0.smithi059.stdout: "nearfull_ratio": 0.85000002384185791, 2020-03-11T22:36:47.899 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_snapshot": "", 2020-03-11T22:36:47.899 INFO:tasks.workunit.client.0.smithi059.stdout: "pool_max": 1, 2020-03-11T22:36:47.899 INFO:tasks.workunit.client.0.smithi059.stdout: "max_osd": 3, 2020-03-11T22:36:47.899 INFO:tasks.workunit.client.0.smithi059.stdout: "require_min_compat_client": "jewel", 2020-03-11T22:36:47.899 INFO:tasks.workunit.client.0.smithi059.stdout: "min_compat_client": "jewel", 2020-03-11T22:36:47.900 INFO:tasks.workunit.client.0.smithi059.stdout: "require_osd_release": "octopus", 2020-03-11T22:36:47.900 INFO:tasks.workunit.client.0.smithi059.stdout: "pools": [ 2020-03-11T22:36:47.900 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:47.900 INFO:tasks.workunit.client.0.smithi059.stdout: "pool": 1, 2020-03-11T22:36:47.900 INFO:tasks.workunit.client.0.smithi059.stdout: "pool_name": "test", 2020-03-11T22:36:47.900 INFO:tasks.workunit.client.0.smithi059.stdout: "create_time": "2020-03-11T22:35:29.665394+0000", 2020-03-11T22:36:47.901 INFO:tasks.workunit.client.0.smithi059.stdout: "flags": 1, 2020-03-11T22:36:47.901 INFO:tasks.workunit.client.0.smithi059.stdout: "flags_names": "hashpspool", 2020-03-11T22:36:47.901 INFO:tasks.workunit.client.0.smithi059.stdout: "type": 1, 2020-03-11T22:36:47.901 INFO:tasks.workunit.client.0.smithi059.stdout: "size": 3, 2020-03-11T22:36:47.901 INFO:tasks.workunit.client.0.smithi059.stdout: "min_size": 2, 2020-03-11T22:36:47.901 INFO:tasks.workunit.client.0.smithi059.stdout: "crush_rule": 0, 2020-03-11T22:36:47.901 INFO:tasks.workunit.client.0.smithi059.stdout: "object_hash": 2, 2020-03-11T22:36:47.902 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_autoscale_mode": "off", 2020-03-11T22:36:47.902 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_num": 1, 2020-03-11T22:36:47.902 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_placement_num": 1, 2020-03-11T22:36:47.902 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_placement_num_target": 1, 2020-03-11T22:36:47.902 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_num_target": 1, 2020-03-11T22:36:47.902 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_num_pending": 1, 2020-03-11T22:36:47.903 INFO:tasks.workunit.client.0.smithi059.stdout: "last_pg_merge_meta": { 2020-03-11T22:36:47.903 INFO:tasks.workunit.client.0.smithi059.stdout: "source_pgid": "0.0", 2020-03-11T22:36:47.903 INFO:tasks.workunit.client.0.smithi059.stdout: "ready_epoch": 0, 2020-03-11T22:36:47.903 INFO:tasks.workunit.client.0.smithi059.stdout: "last_epoch_started": 0, 2020-03-11T22:36:47.903 INFO:tasks.workunit.client.0.smithi059.stdout: "last_epoch_clean": 0, 2020-03-11T22:36:47.903 INFO:tasks.workunit.client.0.smithi059.stdout: "source_version": "0'0", 2020-03-11T22:36:47.904 INFO:tasks.workunit.client.0.smithi059.stdout: "target_version": "0'0" 2020-03-11T22:36:47.904 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:47.904 INFO:tasks.workunit.client.0.smithi059.stdout: "last_change": "28", 2020-03-11T22:36:47.904 INFO:tasks.workunit.client.0.smithi059.stdout: "last_force_op_resend": "0", 2020-03-11T22:36:47.904 INFO:tasks.workunit.client.0.smithi059.stdout: "last_force_op_resend_prenautilus": "0", 2020-03-11T22:36:47.904 INFO:tasks.workunit.client.0.smithi059.stdout: "last_force_op_resend_preluminous": "0", 2020-03-11T22:36:47.904 INFO:tasks.workunit.client.0.smithi059.stdout: "auid": 0, 2020-03-11T22:36:47.905 INFO:tasks.workunit.client.0.smithi059.stdout: "snap_mode": "selfmanaged", 2020-03-11T22:36:47.905 INFO:tasks.workunit.client.0.smithi059.stdout: "snap_seq": 0, 2020-03-11T22:36:47.905 INFO:tasks.workunit.client.0.smithi059.stdout: "snap_epoch": 0, 2020-03-11T22:36:47.905 INFO:tasks.workunit.client.0.smithi059.stdout: "pool_snaps": [], 2020-03-11T22:36:47.905 INFO:tasks.workunit.client.0.smithi059.stdout: "removed_snaps": "[]", 2020-03-11T22:36:47.906 INFO:tasks.workunit.client.0.smithi059.stdout: "quota_max_bytes": 0, 2020-03-11T22:36:47.906 INFO:tasks.workunit.client.0.smithi059.stdout: "quota_max_objects": 0, 2020-03-11T22:36:47.906 INFO:tasks.workunit.client.0.smithi059.stdout: "tiers": [], 2020-03-11T22:36:47.906 INFO:tasks.workunit.client.0.smithi059.stdout: "tier_of": -1, 2020-03-11T22:36:47.906 INFO:tasks.workunit.client.0.smithi059.stdout: "read_tier": -1, 2020-03-11T22:36:47.906 INFO:tasks.workunit.client.0.smithi059.stdout: "write_tier": -1, 2020-03-11T22:36:47.907 INFO:tasks.workunit.client.0.smithi059.stdout: "cache_mode": "none", 2020-03-11T22:36:47.907 INFO:tasks.workunit.client.0.smithi059.stdout: "target_max_bytes": 0, 2020-03-11T22:36:47.907 INFO:tasks.workunit.client.0.smithi059.stdout: "target_max_objects": 0, 2020-03-11T22:36:47.907 INFO:tasks.workunit.client.0.smithi059.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-03-11T22:36:47.907 INFO:tasks.workunit.client.0.smithi059.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-03-11T22:36:47.907 INFO:tasks.workunit.client.0.smithi059.stdout: "cache_target_full_ratio_micro": 800000, 2020-03-11T22:36:47.908 INFO:tasks.workunit.client.0.smithi059.stdout: "cache_min_flush_age": 0, 2020-03-11T22:36:47.908 INFO:tasks.workunit.client.0.smithi059.stdout: "cache_min_evict_age": 0, 2020-03-11T22:36:47.908 INFO:tasks.workunit.client.0.smithi059.stdout: "erasure_code_profile": "", 2020-03-11T22:36:47.908 INFO:tasks.workunit.client.0.smithi059.stdout: "hit_set_params": { 2020-03-11T22:36:47.908 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "none" 2020-03-11T22:36:47.908 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:47.909 INFO:tasks.workunit.client.0.smithi059.stdout: "hit_set_period": 0, 2020-03-11T22:36:47.909 INFO:tasks.workunit.client.0.smithi059.stdout: "hit_set_count": 0, 2020-03-11T22:36:47.909 INFO:tasks.workunit.client.0.smithi059.stdout: "use_gmt_hitset": true, 2020-03-11T22:36:47.909 INFO:tasks.workunit.client.0.smithi059.stdout: "min_read_recency_for_promote": 0, 2020-03-11T22:36:47.909 INFO:tasks.workunit.client.0.smithi059.stdout: "min_write_recency_for_promote": 0, 2020-03-11T22:36:47.909 INFO:tasks.workunit.client.0.smithi059.stdout: "hit_set_grade_decay_rate": 0, 2020-03-11T22:36:47.910 INFO:tasks.workunit.client.0.smithi059.stdout: "hit_set_search_last_n": 0, 2020-03-11T22:36:47.910 INFO:tasks.workunit.client.0.smithi059.stdout: "grade_table": [], 2020-03-11T22:36:47.910 INFO:tasks.workunit.client.0.smithi059.stdout: "stripe_width": 0, 2020-03-11T22:36:47.910 INFO:tasks.workunit.client.0.smithi059.stdout: "expected_num_objects": 0, 2020-03-11T22:36:47.910 INFO:tasks.workunit.client.0.smithi059.stdout: "fast_read": false, 2020-03-11T22:36:47.910 INFO:tasks.workunit.client.0.smithi059.stdout: "options": {}, 2020-03-11T22:36:47.911 INFO:tasks.workunit.client.0.smithi059.stdout: "application_metadata": {} 2020-03-11T22:36:47.911 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:36:47.911 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:36:47.911 INFO:tasks.workunit.client.0.smithi059.stdout: "osds": [ 2020-03-11T22:36:47.911 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:47.911 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 0, 2020-03-11T22:36:47.911 INFO:tasks.workunit.client.0.smithi059.stdout: "uuid": "6b6c1a2d-a55b-410f-9b5c-d5f724d8ff3d", 2020-03-11T22:36:47.912 INFO:tasks.workunit.client.0.smithi059.stdout: "up": 1, 2020-03-11T22:36:47.912 INFO:tasks.workunit.client.0.smithi059.stdout: "in": 1, 2020-03-11T22:36:47.912 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 1, 2020-03-11T22:36:47.912 INFO:tasks.workunit.client.0.smithi059.stdout: "primary_affinity": 1, 2020-03-11T22:36:47.912 INFO:tasks.workunit.client.0.smithi059.stdout: "last_clean_begin": 5, 2020-03-11T22:36:47.912 INFO:tasks.workunit.client.0.smithi059.stdout: "last_clean_end": 28, 2020-03-11T22:36:47.912 INFO:tasks.workunit.client.0.smithi059.stdout: "up_from": 34, 2020-03-11T22:36:47.913 INFO:tasks.workunit.client.0.smithi059.stdout: "up_thru": 39, 2020-03-11T22:36:47.913 INFO:tasks.workunit.client.0.smithi059.stdout: "down_at": 29, 2020-03-11T22:36:47.913 INFO:tasks.workunit.client.0.smithi059.stdout: "lost_at": 0, 2020-03-11T22:36:47.913 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addrs": { 2020-03-11T22:36:47.913 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:36:47.913 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:47.914 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:36:47.914 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6802", 2020-03-11T22:36:47.914 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 30799 2020-03-11T22:36:47.914 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:47.914 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:47.914 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:36:47.914 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6803", 2020-03-11T22:36:47.915 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 30799 2020-03-11T22:36:47.915 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:36:47.915 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:36:47.915 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:47.915 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_addrs": { 2020-03-11T22:36:47.915 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:36:47.916 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:47.916 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:36:47.916 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6804", 2020-03-11T22:36:47.916 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 30799 2020-03-11T22:36:47.916 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:47.916 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:47.917 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:36:47.917 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6805", 2020-03-11T22:36:47.917 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 30799 2020-03-11T22:36:47.917 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:36:47.917 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:36:47.917 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:47.917 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_back_addrs": { 2020-03-11T22:36:47.918 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:36:47.918 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:47.918 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:36:47.918 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6808", 2020-03-11T22:36:47.918 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 30799 2020-03-11T22:36:47.918 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:47.919 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:47.919 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:36:47.919 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6809", 2020-03-11T22:36:47.919 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 30799 2020-03-11T22:36:47.919 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:36:47.919 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:36:47.919 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:47.920 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_front_addrs": { 2020-03-11T22:36:47.920 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:36:47.920 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:47.920 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:36:47.920 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6806", 2020-03-11T22:36:47.920 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 30799 2020-03-11T22:36:47.921 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:47.921 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:47.921 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:36:47.921 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6807", 2020-03-11T22:36:47.921 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 30799 2020-03-11T22:36:47.921 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:36:47.921 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:36:47.922 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:47.922 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addr": "127.0.0.1:6803/30799", 2020-03-11T22:36:47.922 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_addr": "127.0.0.1:6805/30799", 2020-03-11T22:36:47.922 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_back_addr": "127.0.0.1:6809/30799", 2020-03-11T22:36:47.922 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_front_addr": "127.0.0.1:6807/30799", 2020-03-11T22:36:47.922 INFO:tasks.workunit.client.0.smithi059.stdout: "state": [ 2020-03-11T22:36:47.923 INFO:tasks.workunit.client.0.smithi059.stdout: "exists", 2020-03-11T22:36:47.923 INFO:tasks.workunit.client.0.smithi059.stdout: "up" 2020-03-11T22:36:47.923 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:36:47.923 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:47.923 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:47.923 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 1, 2020-03-11T22:36:47.924 INFO:tasks.workunit.client.0.smithi059.stdout: "uuid": "55fb3134-fe56-4d99-9135-8d0c47f417da", 2020-03-11T22:36:47.924 INFO:tasks.workunit.client.0.smithi059.stdout: "up": 0, 2020-03-11T22:36:47.924 INFO:tasks.workunit.client.0.smithi059.stdout: "in": 1, 2020-03-11T22:36:47.924 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 1, 2020-03-11T22:36:47.924 INFO:tasks.workunit.client.0.smithi059.stdout: "primary_affinity": 1, 2020-03-11T22:36:47.924 INFO:tasks.workunit.client.0.smithi059.stdout: "last_clean_begin": 0, 2020-03-11T22:36:47.924 INFO:tasks.workunit.client.0.smithi059.stdout: "last_clean_end": 0, 2020-03-11T22:36:47.925 INFO:tasks.workunit.client.0.smithi059.stdout: "up_from": 10, 2020-03-11T22:36:47.925 INFO:tasks.workunit.client.0.smithi059.stdout: "up_thru": 22, 2020-03-11T22:36:47.925 INFO:tasks.workunit.client.0.smithi059.stdout: "down_at": 30, 2020-03-11T22:36:47.925 INFO:tasks.workunit.client.0.smithi059.stdout: "lost_at": 0, 2020-03-11T22:36:47.925 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addrs": { 2020-03-11T22:36:47.925 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:36:47.926 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:47.926 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:36:47.926 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:36:47.926 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 26394 2020-03-11T22:36:47.926 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:47.926 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:47.926 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:36:47.927 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:36:47.927 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 26394 2020-03-11T22:36:47.927 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:36:47.927 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:36:47.927 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:47.927 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_addrs": { 2020-03-11T22:36:47.927 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:36:47.928 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:47.928 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:36:47.928 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:36:47.928 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 26394 2020-03-11T22:36:47.928 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:47.928 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:47.929 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:36:47.929 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:36:47.929 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 26394 2020-03-11T22:36:47.929 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:36:47.929 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:36:47.929 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:47.930 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_back_addrs": { 2020-03-11T22:36:47.930 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:36:47.930 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:47.930 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:36:47.930 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:36:47.931 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 26394 2020-03-11T22:36:47.931 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:47.931 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:47.931 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:36:47.931 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:36:47.931 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 26394 2020-03-11T22:36:47.932 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:36:47.932 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:36:47.932 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:47.932 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_front_addrs": { 2020-03-11T22:36:47.932 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:36:47.932 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:47.933 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:36:47.933 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:36:47.933 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 26394 2020-03-11T22:36:47.933 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:47.933 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:47.933 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:36:47.934 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:36:47.934 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 26394 2020-03-11T22:36:47.934 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:36:47.934 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:36:47.934 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:47.934 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addr": "127.0.0.1:6811/26394", 2020-03-11T22:36:47.934 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_addr": "127.0.0.1:6813/26394", 2020-03-11T22:36:47.935 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_back_addr": "127.0.0.1:6817/26394", 2020-03-11T22:36:47.935 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_front_addr": "127.0.0.1:6815/26394", 2020-03-11T22:36:47.935 INFO:tasks.workunit.client.0.smithi059.stdout: "state": [ 2020-03-11T22:36:47.935 INFO:tasks.workunit.client.0.smithi059.stdout: "exists" 2020-03-11T22:36:47.935 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:36:47.935 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:47.936 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:47.936 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 2, 2020-03-11T22:36:47.936 INFO:tasks.workunit.client.0.smithi059.stdout: "uuid": "3aa1b594-c731-4355-b2e6-05a8c3d7bb4b", 2020-03-11T22:36:47.936 INFO:tasks.workunit.client.0.smithi059.stdout: "up": 1, 2020-03-11T22:36:47.936 INFO:tasks.workunit.client.0.smithi059.stdout: "in": 1, 2020-03-11T22:36:47.936 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 1, 2020-03-11T22:36:47.936 INFO:tasks.workunit.client.0.smithi059.stdout: "primary_affinity": 1, 2020-03-11T22:36:47.937 INFO:tasks.workunit.client.0.smithi059.stdout: "last_clean_begin": 15, 2020-03-11T22:36:47.937 INFO:tasks.workunit.client.0.smithi059.stdout: "last_clean_end": 28, 2020-03-11T22:36:47.937 INFO:tasks.workunit.client.0.smithi059.stdout: "up_from": 35, 2020-03-11T22:36:47.937 INFO:tasks.workunit.client.0.smithi059.stdout: "up_thru": 0, 2020-03-11T22:36:47.937 INFO:tasks.workunit.client.0.smithi059.stdout: "down_at": 31, 2020-03-11T22:36:47.937 INFO:tasks.workunit.client.0.smithi059.stdout: "lost_at": 0, 2020-03-11T22:36:47.938 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addrs": { 2020-03-11T22:36:47.938 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:36:47.938 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:47.938 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:36:47.938 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:36:47.938 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 30993 2020-03-11T22:36:47.939 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:47.939 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:47.939 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:36:47.939 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:36:47.939 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 30993 2020-03-11T22:36:47.939 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:36:47.939 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:36:47.940 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:47.940 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_addrs": { 2020-03-11T22:36:47.940 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:36:47.940 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:47.940 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:36:47.940 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:36:47.941 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 30993 2020-03-11T22:36:47.941 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:47.941 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:47.941 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:36:47.941 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:36:47.941 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 30993 2020-03-11T22:36:47.941 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:36:47.942 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:36:47.942 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:47.942 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_back_addrs": { 2020-03-11T22:36:47.942 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:36:47.943 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:47.943 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:36:47.943 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:36:47.943 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 30993 2020-03-11T22:36:47.943 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:47.943 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:47.944 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:36:47.944 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:36:47.944 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 30993 2020-03-11T22:36:47.944 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:36:47.944 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:36:47.944 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:47.945 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_front_addrs": { 2020-03-11T22:36:47.945 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:36:47.945 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:47.945 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:36:47.945 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:36:47.945 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 30993 2020-03-11T22:36:47.945 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:47.946 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:47.946 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:36:47.946 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:36:47.946 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 30993 2020-03-11T22:36:47.946 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:36:47.947 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:36:47.947 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:47.947 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addr": "127.0.0.1:6811/30993", 2020-03-11T22:36:47.947 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_addr": "127.0.0.1:6813/30993", 2020-03-11T22:36:47.947 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_back_addr": "127.0.0.1:6817/30993", 2020-03-11T22:36:47.947 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_front_addr": "127.0.0.1:6815/30993", 2020-03-11T22:36:47.947 INFO:tasks.workunit.client.0.smithi059.stdout: "state": [ 2020-03-11T22:36:47.948 INFO:tasks.workunit.client.0.smithi059.stdout: "exists", 2020-03-11T22:36:47.948 INFO:tasks.workunit.client.0.smithi059.stdout: "up" 2020-03-11T22:36:47.948 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:36:47.948 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:36:47.948 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:36:47.948 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_xinfo": [ 2020-03-11T22:36:47.949 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:47.949 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 0, 2020-03-11T22:36:47.949 INFO:tasks.workunit.client.0.smithi059.stdout: "down_stamp": "2020-03-11T22:36:03.766729+0000", 2020-03-11T22:36:47.949 INFO:tasks.workunit.client.0.smithi059.stdout: "laggy_probability": 0, 2020-03-11T22:36:47.949 INFO:tasks.workunit.client.0.smithi059.stdout: "laggy_interval": 0, 2020-03-11T22:36:47.950 INFO:tasks.workunit.client.0.smithi059.stdout: "features": 4540138292836696063, 2020-03-11T22:36:47.950 INFO:tasks.workunit.client.0.smithi059.stdout: "old_weight": 0, 2020-03-11T22:36:47.950 INFO:tasks.workunit.client.0.smithi059.stdout: "last_purged_snaps_scrub": "2020-03-11T22:35:14.607103+0000", 2020-03-11T22:36:47.950 INFO:tasks.workunit.client.0.smithi059.stdout: "dead_epoch": 0 2020-03-11T22:36:47.950 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:47.950 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:47.950 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 1, 2020-03-11T22:36:47.951 INFO:tasks.workunit.client.0.smithi059.stdout: "down_stamp": "2020-03-11T22:36:04.292275+0000", 2020-03-11T22:36:47.951 INFO:tasks.workunit.client.0.smithi059.stdout: "laggy_probability": 0, 2020-03-11T22:36:47.951 INFO:tasks.workunit.client.0.smithi059.stdout: "laggy_interval": 0, 2020-03-11T22:36:47.951 INFO:tasks.workunit.client.0.smithi059.stdout: "features": 4540138292836696063, 2020-03-11T22:36:47.951 INFO:tasks.workunit.client.0.smithi059.stdout: "old_weight": 0, 2020-03-11T22:36:47.951 INFO:tasks.workunit.client.0.smithi059.stdout: "last_purged_snaps_scrub": "2020-03-11T22:35:20.928320+0000", 2020-03-11T22:36:47.952 INFO:tasks.workunit.client.0.smithi059.stdout: "dead_epoch": 0 2020-03-11T22:36:47.952 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:47.952 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:47.952 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 2, 2020-03-11T22:36:47.952 INFO:tasks.workunit.client.0.smithi059.stdout: "down_stamp": "2020-03-11T22:36:04.841346+0000", 2020-03-11T22:36:47.953 INFO:tasks.workunit.client.0.smithi059.stdout: "laggy_probability": 0, 2020-03-11T22:36:47.953 INFO:tasks.workunit.client.0.smithi059.stdout: "laggy_interval": 0, 2020-03-11T22:36:47.953 INFO:tasks.workunit.client.0.smithi059.stdout: "features": 4540138292836696063, 2020-03-11T22:36:47.953 INFO:tasks.workunit.client.0.smithi059.stdout: "old_weight": 0, 2020-03-11T22:36:47.953 INFO:tasks.workunit.client.0.smithi059.stdout: "last_purged_snaps_scrub": "2020-03-11T22:35:27.361869+0000", 2020-03-11T22:36:47.953 INFO:tasks.workunit.client.0.smithi059.stdout: "dead_epoch": 0 2020-03-11T22:36:47.954 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:36:47.954 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:36:47.954 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_upmap": [], 2020-03-11T22:36:47.954 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_upmap_items": [], 2020-03-11T22:36:47.954 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_temp": [], 2020-03-11T22:36:47.955 INFO:tasks.workunit.client.0.smithi059.stdout: "primary_temp": [], 2020-03-11T22:36:47.955 INFO:tasks.workunit.client.0.smithi059.stdout: "blacklist": {}, 2020-03-11T22:36:47.955 INFO:tasks.workunit.client.0.smithi059.stdout: "erasure_code_profiles": { 2020-03-11T22:36:47.955 INFO:tasks.workunit.client.0.smithi059.stdout: "default": { 2020-03-11T22:36:47.955 INFO:tasks.workunit.client.0.smithi059.stdout: "k": "2", 2020-03-11T22:36:47.955 INFO:tasks.workunit.client.0.smithi059.stdout: "m": "2", 2020-03-11T22:36:47.955 INFO:tasks.workunit.client.0.smithi059.stdout: "plugin": "jerasure", 2020-03-11T22:36:47.956 INFO:tasks.workunit.client.0.smithi059.stdout: "technique": "reed_sol_van" 2020-03-11T22:36:47.956 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:36:47.956 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:47.956 INFO:tasks.workunit.client.0.smithi059.stdout: "removed_snaps_queue": [], 2020-03-11T22:36:47.956 INFO:tasks.workunit.client.0.smithi059.stdout: "new_removed_snaps": [], 2020-03-11T22:36:47.957 INFO:tasks.workunit.client.0.smithi059.stdout: "new_purged_snaps": [], 2020-03-11T22:36:47.957 INFO:tasks.workunit.client.0.smithi059.stdout: "crush_node_flags": {}, 2020-03-11T22:36:47.957 INFO:tasks.workunit.client.0.smithi059.stdout: "device_class_flags": {} 2020-03-11T22:36:47.957 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:47.957 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_metadata": [ 2020-03-11T22:36:47.957 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:47.958 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 0, 2020-03-11T22:36:47.958 INFO:tasks.workunit.client.0.smithi059.stdout: "arch": "x86_64", 2020-03-11T22:36:47.958 INFO:tasks.workunit.client.0.smithi059.stdout: "back_addr": "[v2:127.0.0.1:6804/30799,v1:127.0.0.1:6805/30799]", 2020-03-11T22:36:47.958 INFO:tasks.workunit.client.0.smithi059.stdout: "back_iface": "", 2020-03-11T22:36:47.958 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs": "1", 2020-03-11T22:36:47.958 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:36:47.958 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:36:47.959 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:36:47.959 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:36:47.959 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:36:47.959 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:36:47.959 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:36:47.959 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2020-03-11T22:36:47.960 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:36:47.960 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:36:47.960 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:36:47.960 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:36:47.960 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_release": "octopus", 2020-03-11T22:36:47.961 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:36:47.961 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:36:47.961 INFO:tasks.workunit.client.0.smithi059.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-03-11T22:36:47.961 INFO:tasks.workunit.client.0.smithi059.stdout: "default_device_class": "hdd", 2020-03-11T22:36:47.961 INFO:tasks.workunit.client.0.smithi059.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4J2P7", 2020-03-11T22:36:47.961 INFO:tasks.workunit.client.0.smithi059.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:36:47.962 INFO:tasks.workunit.client.0.smithi059.stdout: "devices": "sda", 2020-03-11T22:36:47.962 INFO:tasks.workunit.client.0.smithi059.stdout: "distro": "centos", 2020-03-11T22:36:47.962 INFO:tasks.workunit.client.0.smithi059.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:36:47.962 INFO:tasks.workunit.client.0.smithi059.stdout: "distro_version": "8", 2020-03-11T22:36:47.962 INFO:tasks.workunit.client.0.smithi059.stdout: "front_addr": "[v2:127.0.0.1:6802/30799,v1:127.0.0.1:6803/30799]", 2020-03-11T22:36:47.962 INFO:tasks.workunit.client.0.smithi059.stdout: "front_iface": "", 2020-03-11T22:36:47.963 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/30799,v1:127.0.0.1:6809/30799]", 2020-03-11T22:36:47.963 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/30799,v1:127.0.0.1:6807/30799]", 2020-03-11T22:36:47.963 INFO:tasks.workunit.client.0.smithi059.stdout: "hostname": "smithi059", 2020-03-11T22:36:47.963 INFO:tasks.workunit.client.0.smithi059.stdout: "journal_rotational": "1", 2020-03-11T22:36:47.963 INFO:tasks.workunit.client.0.smithi059.stdout: "kernel_description": "#1 SMP Wed Feb 5 02:00:39 UTC 2020", 2020-03-11T22:36:47.964 INFO:tasks.workunit.client.0.smithi059.stdout: "kernel_version": "4.18.0-147.5.1.el8_1.x86_64", 2020-03-11T22:36:47.964 INFO:tasks.workunit.client.0.smithi059.stdout: "mem_swap_kb": "0", 2020-03-11T22:36:47.964 INFO:tasks.workunit.client.0.smithi059.stdout: "mem_total_kb": "32662084", 2020-03-11T22:36:47.964 INFO:tasks.workunit.client.0.smithi059.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:36:47.964 INFO:tasks.workunit.client.0.smithi059.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:36:47.964 INFO:tasks.workunit.client.0.smithi059.stdout: "os": "Linux", 2020-03-11T22:36:47.964 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_data": "td/divergent-priors/0", 2020-03-11T22:36:47.965 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:36:47.965 INFO:tasks.workunit.client.0.smithi059.stdout: "rotational": "1" 2020-03-11T22:36:47.965 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:47.965 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:47.965 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 1, 2020-03-11T22:36:47.965 INFO:tasks.workunit.client.0.smithi059.stdout: "arch": "x86_64", 2020-03-11T22:36:47.966 INFO:tasks.workunit.client.0.smithi059.stdout: "back_addr": "[v2:127.0.0.1:6812/26394,v1:127.0.0.1:6813/26394]", 2020-03-11T22:36:47.966 INFO:tasks.workunit.client.0.smithi059.stdout: "back_iface": "", 2020-03-11T22:36:47.966 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs": "1", 2020-03-11T22:36:47.966 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:36:47.966 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:36:47.966 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:36:47.967 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:36:47.967 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:36:47.967 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:36:47.967 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:36:47.967 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2020-03-11T22:36:47.968 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:36:47.968 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:36:47.968 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:36:47.968 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:36:47.968 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_release": "octopus", 2020-03-11T22:36:47.968 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:36:47.968 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:36:47.969 INFO:tasks.workunit.client.0.smithi059.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-03-11T22:36:47.969 INFO:tasks.workunit.client.0.smithi059.stdout: "default_device_class": "hdd", 2020-03-11T22:36:47.969 INFO:tasks.workunit.client.0.smithi059.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4J2P7", 2020-03-11T22:36:47.969 INFO:tasks.workunit.client.0.smithi059.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:36:47.969 INFO:tasks.workunit.client.0.smithi059.stdout: "devices": "sda", 2020-03-11T22:36:47.969 INFO:tasks.workunit.client.0.smithi059.stdout: "distro": "centos", 2020-03-11T22:36:47.970 INFO:tasks.workunit.client.0.smithi059.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:36:47.970 INFO:tasks.workunit.client.0.smithi059.stdout: "distro_version": "8", 2020-03-11T22:36:47.970 INFO:tasks.workunit.client.0.smithi059.stdout: "front_addr": "[v2:127.0.0.1:6810/26394,v1:127.0.0.1:6811/26394]", 2020-03-11T22:36:47.970 INFO:tasks.workunit.client.0.smithi059.stdout: "front_iface": "", 2020-03-11T22:36:47.970 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/26394,v1:127.0.0.1:6817/26394]", 2020-03-11T22:36:47.970 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/26394,v1:127.0.0.1:6815/26394]", 2020-03-11T22:36:47.971 INFO:tasks.workunit.client.0.smithi059.stdout: "hostname": "smithi059", 2020-03-11T22:36:47.971 INFO:tasks.workunit.client.0.smithi059.stdout: "journal_rotational": "1", 2020-03-11T22:36:47.971 INFO:tasks.workunit.client.0.smithi059.stdout: "kernel_description": "#1 SMP Wed Feb 5 02:00:39 UTC 2020", 2020-03-11T22:36:47.971 INFO:tasks.workunit.client.0.smithi059.stdout: "kernel_version": "4.18.0-147.5.1.el8_1.x86_64", 2020-03-11T22:36:47.971 INFO:tasks.workunit.client.0.smithi059.stdout: "mem_swap_kb": "0", 2020-03-11T22:36:47.971 INFO:tasks.workunit.client.0.smithi059.stdout: "mem_total_kb": "32662084", 2020-03-11T22:36:47.971 INFO:tasks.workunit.client.0.smithi059.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:36:47.972 INFO:tasks.workunit.client.0.smithi059.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:36:47.972 INFO:tasks.workunit.client.0.smithi059.stdout: "os": "Linux", 2020-03-11T22:36:47.972 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_data": "td/divergent-priors/1", 2020-03-11T22:36:47.972 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:36:47.972 INFO:tasks.workunit.client.0.smithi059.stdout: "rotational": "1" 2020-03-11T22:36:47.972 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:47.973 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:47.973 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 2, 2020-03-11T22:36:47.973 INFO:tasks.workunit.client.0.smithi059.stdout: "arch": "x86_64", 2020-03-11T22:36:47.973 INFO:tasks.workunit.client.0.smithi059.stdout: "back_addr": "[v2:127.0.0.1:6812/30993,v1:127.0.0.1:6813/30993]", 2020-03-11T22:36:47.973 INFO:tasks.workunit.client.0.smithi059.stdout: "back_iface": "", 2020-03-11T22:36:47.974 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs": "1", 2020-03-11T22:36:47.974 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:36:47.974 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:36:47.974 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:36:47.974 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:36:47.974 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:36:47.974 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:36:47.975 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:36:47.975 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2020-03-11T22:36:47.975 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:36:47.975 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:36:47.975 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:36:47.975 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:36:47.976 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_release": "octopus", 2020-03-11T22:36:47.976 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:36:47.976 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:36:47.976 INFO:tasks.workunit.client.0.smithi059.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-03-11T22:36:47.976 INFO:tasks.workunit.client.0.smithi059.stdout: "default_device_class": "hdd", 2020-03-11T22:36:47.977 INFO:tasks.workunit.client.0.smithi059.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4J2P7", 2020-03-11T22:36:47.977 INFO:tasks.workunit.client.0.smithi059.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:36:47.977 INFO:tasks.workunit.client.0.smithi059.stdout: "devices": "sda", 2020-03-11T22:36:47.977 INFO:tasks.workunit.client.0.smithi059.stdout: "distro": "centos", 2020-03-11T22:36:47.977 INFO:tasks.workunit.client.0.smithi059.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:36:47.977 INFO:tasks.workunit.client.0.smithi059.stdout: "distro_version": "8", 2020-03-11T22:36:47.977 INFO:tasks.workunit.client.0.smithi059.stdout: "front_addr": "[v2:127.0.0.1:6810/30993,v1:127.0.0.1:6811/30993]", 2020-03-11T22:36:47.978 INFO:tasks.workunit.client.0.smithi059.stdout: "front_iface": "", 2020-03-11T22:36:47.978 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/30993,v1:127.0.0.1:6817/30993]", 2020-03-11T22:36:47.978 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/30993,v1:127.0.0.1:6815/30993]", 2020-03-11T22:36:47.978 INFO:tasks.workunit.client.0.smithi059.stdout: "hostname": "smithi059", 2020-03-11T22:36:47.978 INFO:tasks.workunit.client.0.smithi059.stdout: "journal_rotational": "1", 2020-03-11T22:36:47.978 INFO:tasks.workunit.client.0.smithi059.stdout: "kernel_description": "#1 SMP Wed Feb 5 02:00:39 UTC 2020", 2020-03-11T22:36:47.979 INFO:tasks.workunit.client.0.smithi059.stdout: "kernel_version": "4.18.0-147.5.1.el8_1.x86_64", 2020-03-11T22:36:47.979 INFO:tasks.workunit.client.0.smithi059.stdout: "mem_swap_kb": "0", 2020-03-11T22:36:47.979 INFO:tasks.workunit.client.0.smithi059.stdout: "mem_total_kb": "32662084", 2020-03-11T22:36:47.979 INFO:tasks.workunit.client.0.smithi059.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:36:47.979 INFO:tasks.workunit.client.0.smithi059.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:36:47.979 INFO:tasks.workunit.client.0.smithi059.stdout: "os": "Linux", 2020-03-11T22:36:47.979 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_data": "td/divergent-priors/2", 2020-03-11T22:36:47.980 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:36:47.980 INFO:tasks.workunit.client.0.smithi059.stdout: "rotational": "1" 2020-03-11T22:36:47.980 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:36:47.980 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:36:47.980 INFO:tasks.workunit.client.0.smithi059.stdout: "osdmap_first_committed": 1, 2020-03-11T22:36:47.980 INFO:tasks.workunit.client.0.smithi059.stdout: "osdmap_last_committed": 40, 2020-03-11T22:36:47.981 INFO:tasks.workunit.client.0.smithi059.stdout: "crushmap": { 2020-03-11T22:36:47.981 INFO:tasks.workunit.client.0.smithi059.stdout: "devices": [ 2020-03-11T22:36:47.981 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:47.981 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 0, 2020-03-11T22:36:47.981 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "osd.0", 2020-03-11T22:36:47.981 INFO:tasks.workunit.client.0.smithi059.stdout: "class": "hdd" 2020-03-11T22:36:47.982 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:47.982 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:47.982 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 1, 2020-03-11T22:36:47.982 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "osd.1", 2020-03-11T22:36:47.982 INFO:tasks.workunit.client.0.smithi059.stdout: "class": "hdd" 2020-03-11T22:36:47.982 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:47.983 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:47.983 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 2, 2020-03-11T22:36:47.983 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "osd.2", 2020-03-11T22:36:47.983 INFO:tasks.workunit.client.0.smithi059.stdout: "class": "hdd" 2020-03-11T22:36:47.983 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:36:47.983 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:36:47.984 INFO:tasks.workunit.client.0.smithi059.stdout: "types": [ 2020-03-11T22:36:47.984 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:47.984 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 0, 2020-03-11T22:36:47.984 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "osd" 2020-03-11T22:36:47.984 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:47.984 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:47.985 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 1, 2020-03-11T22:36:47.985 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "host" 2020-03-11T22:36:47.985 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:47.985 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:47.985 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 2, 2020-03-11T22:36:47.985 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "chassis" 2020-03-11T22:36:47.986 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:47.986 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:47.986 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 3, 2020-03-11T22:36:47.986 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "rack" 2020-03-11T22:36:47.986 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:47.986 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:47.987 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 4, 2020-03-11T22:36:47.987 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "row" 2020-03-11T22:36:47.987 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:47.987 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:47.987 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 5, 2020-03-11T22:36:47.987 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "pdu" 2020-03-11T22:36:47.987 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:47.988 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:47.988 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 6, 2020-03-11T22:36:47.988 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "pod" 2020-03-11T22:36:47.988 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:47.988 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:47.988 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 7, 2020-03-11T22:36:47.989 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "room" 2020-03-11T22:36:47.989 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:47.989 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:47.989 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 8, 2020-03-11T22:36:47.989 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "datacenter" 2020-03-11T22:36:47.989 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:47.990 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:47.990 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 9, 2020-03-11T22:36:47.990 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "zone" 2020-03-11T22:36:47.990 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:47.990 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:47.991 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 10, 2020-03-11T22:36:47.991 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "region" 2020-03-11T22:36:47.991 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:47.991 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:47.991 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 11, 2020-03-11T22:36:47.991 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "root" 2020-03-11T22:36:47.992 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:36:47.992 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:36:47.992 INFO:tasks.workunit.client.0.smithi059.stdout: "buckets": [ 2020-03-11T22:36:47.992 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:47.992 INFO:tasks.workunit.client.0.smithi059.stdout: "id": -1, 2020-03-11T22:36:47.992 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "default", 2020-03-11T22:36:47.993 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 11, 2020-03-11T22:36:47.993 INFO:tasks.workunit.client.0.smithi059.stdout: "type_name": "root", 2020-03-11T22:36:47.993 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 19206, 2020-03-11T22:36:47.993 INFO:tasks.workunit.client.0.smithi059.stdout: "alg": "straw2", 2020-03-11T22:36:47.993 INFO:tasks.workunit.client.0.smithi059.stdout: "hash": "rjenkins1", 2020-03-11T22:36:47.993 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2020-03-11T22:36:47.994 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:47.994 INFO:tasks.workunit.client.0.smithi059.stdout: "id": -3, 2020-03-11T22:36:47.994 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 19206, 2020-03-11T22:36:47.994 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 0 2020-03-11T22:36:47.994 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:36:47.994 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:36:47.994 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:47.995 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:47.995 INFO:tasks.workunit.client.0.smithi059.stdout: "id": -2, 2020-03-11T22:36:47.995 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "default~hdd", 2020-03-11T22:36:47.995 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 11, 2020-03-11T22:36:47.995 INFO:tasks.workunit.client.0.smithi059.stdout: "type_name": "root", 2020-03-11T22:36:47.995 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 19206, 2020-03-11T22:36:47.996 INFO:tasks.workunit.client.0.smithi059.stdout: "alg": "straw2", 2020-03-11T22:36:47.996 INFO:tasks.workunit.client.0.smithi059.stdout: "hash": "rjenkins1", 2020-03-11T22:36:47.996 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2020-03-11T22:36:47.996 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:47.996 INFO:tasks.workunit.client.0.smithi059.stdout: "id": -4, 2020-03-11T22:36:47.996 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 19206, 2020-03-11T22:36:47.996 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 0 2020-03-11T22:36:47.997 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:36:47.997 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:36:47.997 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:47.997 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:47.997 INFO:tasks.workunit.client.0.smithi059.stdout: "id": -3, 2020-03-11T22:36:48.000 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "smithi059", 2020-03-11T22:36:48.001 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 1, 2020-03-11T22:36:48.001 INFO:tasks.workunit.client.0.smithi059.stdout: "type_name": "host", 2020-03-11T22:36:48.001 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 19206, 2020-03-11T22:36:48.001 INFO:tasks.workunit.client.0.smithi059.stdout: "alg": "straw2", 2020-03-11T22:36:48.001 INFO:tasks.workunit.client.0.smithi059.stdout: "hash": "rjenkins1", 2020-03-11T22:36:48.001 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2020-03-11T22:36:48.001 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:48.002 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 0, 2020-03-11T22:36:48.002 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 6402, 2020-03-11T22:36:48.002 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 0 2020-03-11T22:36:48.002 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:48.002 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:48.002 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 1, 2020-03-11T22:36:48.003 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 6402, 2020-03-11T22:36:48.003 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 1 2020-03-11T22:36:48.003 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:48.003 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:48.003 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 2, 2020-03-11T22:36:48.003 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 6402, 2020-03-11T22:36:48.004 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 2 2020-03-11T22:36:48.004 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:36:48.004 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:36:48.004 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:48.004 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:48.004 INFO:tasks.workunit.client.0.smithi059.stdout: "id": -4, 2020-03-11T22:36:48.005 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "smithi059~hdd", 2020-03-11T22:36:48.005 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 1, 2020-03-11T22:36:48.005 INFO:tasks.workunit.client.0.smithi059.stdout: "type_name": "host", 2020-03-11T22:36:48.005 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 19206, 2020-03-11T22:36:48.005 INFO:tasks.workunit.client.0.smithi059.stdout: "alg": "straw2", 2020-03-11T22:36:48.005 INFO:tasks.workunit.client.0.smithi059.stdout: "hash": "rjenkins1", 2020-03-11T22:36:48.006 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2020-03-11T22:36:48.006 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:48.006 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 0, 2020-03-11T22:36:48.006 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 6402, 2020-03-11T22:36:48.006 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 0 2020-03-11T22:36:48.006 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:48.007 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:48.007 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 1, 2020-03-11T22:36:48.007 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 6402, 2020-03-11T22:36:48.007 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 1 2020-03-11T22:36:48.007 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:48.007 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:48.008 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 2, 2020-03-11T22:36:48.008 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 6402, 2020-03-11T22:36:48.008 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 2 2020-03-11T22:36:48.008 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:36:48.008 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:36:48.008 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:36:48.009 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:36:48.009 INFO:tasks.workunit.client.0.smithi059.stdout: "rules": [ 2020-03-11T22:36:48.009 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:48.009 INFO:tasks.workunit.client.0.smithi059.stdout: "rule_id": 0, 2020-03-11T22:36:48.009 INFO:tasks.workunit.client.0.smithi059.stdout: "rule_name": "replicated_rule", 2020-03-11T22:36:48.009 INFO:tasks.workunit.client.0.smithi059.stdout: "ruleset": 0, 2020-03-11T22:36:48.010 INFO:tasks.workunit.client.0.smithi059.stdout: "type": 1, 2020-03-11T22:36:48.010 INFO:tasks.workunit.client.0.smithi059.stdout: "min_size": 1, 2020-03-11T22:36:48.010 INFO:tasks.workunit.client.0.smithi059.stdout: "max_size": 10, 2020-03-11T22:36:48.010 INFO:tasks.workunit.client.0.smithi059.stdout: "steps": [ 2020-03-11T22:36:48.010 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:48.010 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "take", 2020-03-11T22:36:48.011 INFO:tasks.workunit.client.0.smithi059.stdout: "item": -1, 2020-03-11T22:36:48.011 INFO:tasks.workunit.client.0.smithi059.stdout: "item_name": "default" 2020-03-11T22:36:48.011 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:48.011 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:48.011 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "choose_firstn", 2020-03-11T22:36:48.011 INFO:tasks.workunit.client.0.smithi059.stdout: "num": 0, 2020-03-11T22:36:48.012 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "osd" 2020-03-11T22:36:48.012 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:48.012 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:48.012 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "emit" 2020-03-11T22:36:48.012 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:36:48.012 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:36:48.012 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:36:48.013 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:36:48.013 INFO:tasks.workunit.client.0.smithi059.stdout: "tunables": { 2020-03-11T22:36:48.013 INFO:tasks.workunit.client.0.smithi059.stdout: "choose_local_tries": 0, 2020-03-11T22:36:48.013 INFO:tasks.workunit.client.0.smithi059.stdout: "choose_local_fallback_tries": 0, 2020-03-11T22:36:48.013 INFO:tasks.workunit.client.0.smithi059.stdout: "choose_total_tries": 50, 2020-03-11T22:36:48.013 INFO:tasks.workunit.client.0.smithi059.stdout: "chooseleaf_descend_once": 1, 2020-03-11T22:36:48.014 INFO:tasks.workunit.client.0.smithi059.stdout: "chooseleaf_vary_r": 1, 2020-03-11T22:36:48.014 INFO:tasks.workunit.client.0.smithi059.stdout: "chooseleaf_stable": 1, 2020-03-11T22:36:48.014 INFO:tasks.workunit.client.0.smithi059.stdout: "straw_calc_version": 1, 2020-03-11T22:36:48.014 INFO:tasks.workunit.client.0.smithi059.stdout: "allowed_bucket_algs": 54, 2020-03-11T22:36:48.014 INFO:tasks.workunit.client.0.smithi059.stdout: "profile": "jewel", 2020-03-11T22:36:48.014 INFO:tasks.workunit.client.0.smithi059.stdout: "optimal_tunables": 1, 2020-03-11T22:36:48.014 INFO:tasks.workunit.client.0.smithi059.stdout: "legacy_tunables": 0, 2020-03-11T22:36:48.015 INFO:tasks.workunit.client.0.smithi059.stdout: "minimum_required_version": "jewel", 2020-03-11T22:36:48.015 INFO:tasks.workunit.client.0.smithi059.stdout: "require_feature_tunables": 1, 2020-03-11T22:36:48.015 INFO:tasks.workunit.client.0.smithi059.stdout: "require_feature_tunables2": 1, 2020-03-11T22:36:48.015 INFO:tasks.workunit.client.0.smithi059.stdout: "has_v2_rules": 0, 2020-03-11T22:36:48.015 INFO:tasks.workunit.client.0.smithi059.stdout: "require_feature_tunables3": 1, 2020-03-11T22:36:48.015 INFO:tasks.workunit.client.0.smithi059.stdout: "has_v3_rules": 0, 2020-03-11T22:36:48.016 INFO:tasks.workunit.client.0.smithi059.stdout: "has_v4_buckets": 1, 2020-03-11T22:36:48.016 INFO:tasks.workunit.client.0.smithi059.stdout: "require_feature_tunables5": 1, 2020-03-11T22:36:48.016 INFO:tasks.workunit.client.0.smithi059.stdout: "has_v5_rules": 0 2020-03-11T22:36:48.016 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:48.016 INFO:tasks.workunit.client.0.smithi059.stdout: "choose_args": {} 2020-03-11T22:36:48.016 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:48.016 INFO:tasks.workunit.client.0.smithi059.stdout: "fsmap": { 2020-03-11T22:36:48.016 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": 1, 2020-03-11T22:36:48.017 INFO:tasks.workunit.client.0.smithi059.stdout: "default_fscid": -1, 2020-03-11T22:36:48.017 INFO:tasks.workunit.client.0.smithi059.stdout: "compat": { 2020-03-11T22:36:48.017 INFO:tasks.workunit.client.0.smithi059.stdout: "compat": {}, 2020-03-11T22:36:48.017 INFO:tasks.workunit.client.0.smithi059.stdout: "ro_compat": {}, 2020-03-11T22:36:48.017 INFO:tasks.workunit.client.0.smithi059.stdout: "incompat": { 2020-03-11T22:36:48.017 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_1": "base v0.20", 2020-03-11T22:36:48.018 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_2": "client writeable ranges", 2020-03-11T22:36:48.018 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_3": "default file layouts on dirs", 2020-03-11T22:36:48.018 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_4": "dir inode in separate object", 2020-03-11T22:36:48.018 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_5": "mds uses versioned encoding", 2020-03-11T22:36:48.018 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_6": "dirfrag is stored in omap", 2020-03-11T22:36:48.018 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_8": "no anchor table", 2020-03-11T22:36:48.019 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_9": "file layout v2", 2020-03-11T22:36:48.019 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_10": "snaprealm v2" 2020-03-11T22:36:48.019 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:36:48.019 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:48.019 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_flags": { 2020-03-11T22:36:48.019 INFO:tasks.workunit.client.0.smithi059.stdout: "enable_multiple": false, 2020-03-11T22:36:48.019 INFO:tasks.workunit.client.0.smithi059.stdout: "ever_enabled_multiple": false 2020-03-11T22:36:48.020 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:48.020 INFO:tasks.workunit.client.0.smithi059.stdout: "standbys": [], 2020-03-11T22:36:48.020 INFO:tasks.workunit.client.0.smithi059.stdout: "filesystems": [] 2020-03-11T22:36:48.020 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:48.020 INFO:tasks.workunit.client.0.smithi059.stdout: "mdsmap_first_committed": 1, 2020-03-11T22:36:48.020 INFO:tasks.workunit.client.0.smithi059.stdout: "mdsmap_last_committed": 1, 2020-03-11T22:36:48.020 INFO:tasks.workunit.client.0.smithi059.stdout: "auth": { 2020-03-11T22:36:48.021 INFO:tasks.workunit.client.0.smithi059.stdout: "first_committed": 1, 2020-03-11T22:36:48.021 INFO:tasks.workunit.client.0.smithi059.stdout: "last_committed": 5, 2020-03-11T22:36:48.021 INFO:tasks.workunit.client.0.smithi059.stdout: "num_secrets": 3 2020-03-11T22:36:48.021 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:48.021 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pg": 1, 2020-03-11T22:36:48.021 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pg_active": 1, 2020-03-11T22:36:48.022 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pg_unknown": 0, 2020-03-11T22:36:48.022 INFO:tasks.workunit.client.0.smithi059.stdout: "num_osd": 3, 2020-03-11T22:36:48.022 INFO:tasks.workunit.client.0.smithi059.stdout: "pool_sum": { 2020-03-11T22:36:48.022 INFO:tasks.workunit.client.0.smithi059.stdout: "stat_sum": { 2020-03-11T22:36:48.022 INFO:tasks.workunit.client.0.smithi059.stdout: "num_bytes": 48300, 2020-03-11T22:36:48.022 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects": 100, 2020-03-11T22:36:48.022 INFO:tasks.workunit.client.0.smithi059.stdout: "num_object_clones": 0, 2020-03-11T22:36:48.023 INFO:tasks.workunit.client.0.smithi059.stdout: "num_object_copies": 300, 2020-03-11T22:36:48.023 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:36:48.023 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_missing": 0, 2020-03-11T22:36:48.023 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_degraded": 100, 2020-03-11T22:36:48.023 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_misplaced": 0, 2020-03-11T22:36:48.023 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_unfound": 0, 2020-03-11T22:36:48.024 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_dirty": 100, 2020-03-11T22:36:48.024 INFO:tasks.workunit.client.0.smithi059.stdout: "num_whiteouts": 0, 2020-03-11T22:36:48.024 INFO:tasks.workunit.client.0.smithi059.stdout: "num_read": 0, 2020-03-11T22:36:48.024 INFO:tasks.workunit.client.0.smithi059.stdout: "num_read_kb": 0, 2020-03-11T22:36:48.024 INFO:tasks.workunit.client.0.smithi059.stdout: "num_write": 100, 2020-03-11T22:36:48.024 INFO:tasks.workunit.client.0.smithi059.stdout: "num_write_kb": 100, 2020-03-11T22:36:48.025 INFO:tasks.workunit.client.0.smithi059.stdout: "num_scrub_errors": 0, 2020-03-11T22:36:48.025 INFO:tasks.workunit.client.0.smithi059.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:36:48.025 INFO:tasks.workunit.client.0.smithi059.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:36:48.025 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_recovered": 0, 2020-03-11T22:36:48.025 INFO:tasks.workunit.client.0.smithi059.stdout: "num_bytes_recovered": 0, 2020-03-11T22:36:48.025 INFO:tasks.workunit.client.0.smithi059.stdout: "num_keys_recovered": 0, 2020-03-11T22:36:48.025 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_omap": 0, 2020-03-11T22:36:48.026 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:36:48.026 INFO:tasks.workunit.client.0.smithi059.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:36:48.026 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush": 0, 2020-03-11T22:36:48.026 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush_kb": 0, 2020-03-11T22:36:48.026 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict": 0, 2020-03-11T22:36:48.026 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict_kb": 0, 2020-03-11T22:36:48.027 INFO:tasks.workunit.client.0.smithi059.stdout: "num_promote": 0, 2020-03-11T22:36:48.027 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush_mode_high": 0, 2020-03-11T22:36:48.027 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush_mode_low": 0, 2020-03-11T22:36:48.027 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict_mode_some": 0, 2020-03-11T22:36:48.027 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict_mode_full": 0, 2020-03-11T22:36:48.027 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_pinned": 0, 2020-03-11T22:36:48.027 INFO:tasks.workunit.client.0.smithi059.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:36:48.028 INFO:tasks.workunit.client.0.smithi059.stdout: "num_large_omap_objects": 0, 2020-03-11T22:36:48.028 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_manifest": 0, 2020-03-11T22:36:48.028 INFO:tasks.workunit.client.0.smithi059.stdout: "num_omap_bytes": 0, 2020-03-11T22:36:48.028 INFO:tasks.workunit.client.0.smithi059.stdout: "num_omap_keys": 0, 2020-03-11T22:36:48.028 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_repaired": 0 2020-03-11T22:36:48.028 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:48.029 INFO:tasks.workunit.client.0.smithi059.stdout: "store_stats": { 2020-03-11T22:36:48.029 INFO:tasks.workunit.client.0.smithi059.stdout: "total": 0, 2020-03-11T22:36:48.029 INFO:tasks.workunit.client.0.smithi059.stdout: "available": 0, 2020-03-11T22:36:48.029 INFO:tasks.workunit.client.0.smithi059.stdout: "internally_reserved": 0, 2020-03-11T22:36:48.029 INFO:tasks.workunit.client.0.smithi059.stdout: "allocated": 0, 2020-03-11T22:36:48.030 INFO:tasks.workunit.client.0.smithi059.stdout: "data_stored": 0, 2020-03-11T22:36:48.030 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed": 0, 2020-03-11T22:36:48.030 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_allocated": 0, 2020-03-11T22:36:48.030 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_original": 0, 2020-03-11T22:36:48.030 INFO:tasks.workunit.client.0.smithi059.stdout: "omap_allocated": 0, 2020-03-11T22:36:48.030 INFO:tasks.workunit.client.0.smithi059.stdout: "internal_metadata": 0 2020-03-11T22:36:48.031 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:48.031 INFO:tasks.workunit.client.0.smithi059.stdout: "log_size": 16, 2020-03-11T22:36:48.031 INFO:tasks.workunit.client.0.smithi059.stdout: "ondisk_log_size": 16, 2020-03-11T22:36:48.031 INFO:tasks.workunit.client.0.smithi059.stdout: "up": 2, 2020-03-11T22:36:48.031 INFO:tasks.workunit.client.0.smithi059.stdout: "acting": 2, 2020-03-11T22:36:48.031 INFO:tasks.workunit.client.0.smithi059.stdout: "num_store_stats": 0 2020-03-11T22:36:48.032 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:48.032 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_sum": { 2020-03-11T22:36:48.032 INFO:tasks.workunit.client.0.smithi059.stdout: "up_from": 0, 2020-03-11T22:36:48.032 INFO:tasks.workunit.client.0.smithi059.stdout: "seq": 0, 2020-03-11T22:36:48.032 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pgs": 3, 2020-03-11T22:36:48.032 INFO:tasks.workunit.client.0.smithi059.stdout: "num_osds": 3, 2020-03-11T22:36:48.033 INFO:tasks.workunit.client.0.smithi059.stdout: "num_per_pool_osds": 3, 2020-03-11T22:36:48.033 INFO:tasks.workunit.client.0.smithi059.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:36:48.033 INFO:tasks.workunit.client.0.smithi059.stdout: "kb": 314572800, 2020-03-11T22:36:48.033 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used": 3179328, 2020-03-11T22:36:48.033 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used_data": 33280, 2020-03-11T22:36:48.033 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used_omap": 13, 2020-03-11T22:36:48.034 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used_meta": 3145714, 2020-03-11T22:36:48.034 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_avail": 311393472, 2020-03-11T22:36:48.034 INFO:tasks.workunit.client.0.smithi059.stdout: "statfs": { 2020-03-11T22:36:48.034 INFO:tasks.workunit.client.0.smithi059.stdout: "total": 322122547200, 2020-03-11T22:36:48.034 INFO:tasks.workunit.client.0.smithi059.stdout: "available": 318866915328, 2020-03-11T22:36:48.035 INFO:tasks.workunit.client.0.smithi059.stdout: "internally_reserved": 3221225472, 2020-03-11T22:36:48.035 INFO:tasks.workunit.client.0.smithi059.stdout: "allocated": 34078720, 2020-03-11T22:36:48.035 INFO:tasks.workunit.client.0.smithi059.stdout: "data_stored": 407206, 2020-03-11T22:36:48.035 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed": 0, 2020-03-11T22:36:48.035 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_allocated": 0, 2020-03-11T22:36:48.035 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_original": 0, 2020-03-11T22:36:48.035 INFO:tasks.workunit.client.0.smithi059.stdout: "omap_allocated": 14150, 2020-03-11T22:36:48.036 INFO:tasks.workunit.client.0.smithi059.stdout: "internal_metadata": 3221211322 2020-03-11T22:36:48.036 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:48.036 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_peers": [], 2020-03-11T22:36:48.036 INFO:tasks.workunit.client.0.smithi059.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:36:48.036 INFO:tasks.workunit.client.0.smithi059.stdout: "num_snap_trimming": 0, 2020-03-11T22:36:48.036 INFO:tasks.workunit.client.0.smithi059.stdout: "num_shards_repaired": 0, 2020-03-11T22:36:48.037 INFO:tasks.workunit.client.0.smithi059.stdout: "op_queue_age_hist": { 2020-03-11T22:36:48.037 INFO:tasks.workunit.client.0.smithi059.stdout: "histogram": [], 2020-03-11T22:36:48.037 INFO:tasks.workunit.client.0.smithi059.stdout: "upper_bound": 1 2020-03-11T22:36:48.037 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:48.037 INFO:tasks.workunit.client.0.smithi059.stdout: "perf_stat": { 2020-03-11T22:36:48.037 INFO:tasks.workunit.client.0.smithi059.stdout: "commit_latency_ms": 0, 2020-03-11T22:36:48.038 INFO:tasks.workunit.client.0.smithi059.stdout: "apply_latency_ms": 0, 2020-03-11T22:36:48.038 INFO:tasks.workunit.client.0.smithi059.stdout: "commit_latency_ns": 0, 2020-03-11T22:36:48.038 INFO:tasks.workunit.client.0.smithi059.stdout: "apply_latency_ns": 0 2020-03-11T22:36:48.038 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:48.038 INFO:tasks.workunit.client.0.smithi059.stdout: "alerts": [], 2020-03-11T22:36:48.038 INFO:tasks.workunit.client.0.smithi059.stdout: "network_ping_times": [] 2020-03-11T22:36:48.039 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:48.039 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_sum_by_class": { 2020-03-11T22:36:48.039 INFO:tasks.workunit.client.0.smithi059.stdout: "hdd": { 2020-03-11T22:36:48.039 INFO:tasks.workunit.client.0.smithi059.stdout: "up_from": 0, 2020-03-11T22:36:48.040 INFO:tasks.workunit.client.0.smithi059.stdout: "seq": 0, 2020-03-11T22:36:48.040 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pgs": 3, 2020-03-11T22:36:48.040 INFO:tasks.workunit.client.0.smithi059.stdout: "num_osds": 3, 2020-03-11T22:36:48.040 INFO:tasks.workunit.client.0.smithi059.stdout: "num_per_pool_osds": 3, 2020-03-11T22:36:48.040 INFO:tasks.workunit.client.0.smithi059.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:36:48.040 INFO:tasks.workunit.client.0.smithi059.stdout: "kb": 314572800, 2020-03-11T22:36:48.041 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used": 3179328, 2020-03-11T22:36:48.041 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used_data": 33280, 2020-03-11T22:36:48.041 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used_omap": 13, 2020-03-11T22:36:48.041 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used_meta": 3145714, 2020-03-11T22:36:48.041 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_avail": 311393472, 2020-03-11T22:36:48.041 INFO:tasks.workunit.client.0.smithi059.stdout: "statfs": { 2020-03-11T22:36:48.042 INFO:tasks.workunit.client.0.smithi059.stdout: "total": 322122547200, 2020-03-11T22:36:48.042 INFO:tasks.workunit.client.0.smithi059.stdout: "available": 318866915328, 2020-03-11T22:36:48.042 INFO:tasks.workunit.client.0.smithi059.stdout: "internally_reserved": 3221225472, 2020-03-11T22:36:48.042 INFO:tasks.workunit.client.0.smithi059.stdout: "allocated": 34078720, 2020-03-11T22:36:48.042 INFO:tasks.workunit.client.0.smithi059.stdout: "data_stored": 407206, 2020-03-11T22:36:48.043 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed": 0, 2020-03-11T22:36:48.043 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_allocated": 0, 2020-03-11T22:36:48.043 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_original": 0, 2020-03-11T22:36:48.043 INFO:tasks.workunit.client.0.smithi059.stdout: "omap_allocated": 14150, 2020-03-11T22:36:48.043 INFO:tasks.workunit.client.0.smithi059.stdout: "internal_metadata": 3221211322 2020-03-11T22:36:48.043 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:48.044 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_peers": [], 2020-03-11T22:36:48.044 INFO:tasks.workunit.client.0.smithi059.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:36:48.044 INFO:tasks.workunit.client.0.smithi059.stdout: "num_snap_trimming": 0, 2020-03-11T22:36:48.044 INFO:tasks.workunit.client.0.smithi059.stdout: "num_shards_repaired": 0, 2020-03-11T22:36:48.044 INFO:tasks.workunit.client.0.smithi059.stdout: "op_queue_age_hist": { 2020-03-11T22:36:48.044 INFO:tasks.workunit.client.0.smithi059.stdout: "histogram": [], 2020-03-11T22:36:48.044 INFO:tasks.workunit.client.0.smithi059.stdout: "upper_bound": 1 2020-03-11T22:36:48.045 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:48.045 INFO:tasks.workunit.client.0.smithi059.stdout: "perf_stat": { 2020-03-11T22:36:48.045 INFO:tasks.workunit.client.0.smithi059.stdout: "commit_latency_ms": 0, 2020-03-11T22:36:48.045 INFO:tasks.workunit.client.0.smithi059.stdout: "apply_latency_ms": 0, 2020-03-11T22:36:48.045 INFO:tasks.workunit.client.0.smithi059.stdout: "commit_latency_ns": 0, 2020-03-11T22:36:48.045 INFO:tasks.workunit.client.0.smithi059.stdout: "apply_latency_ns": 0 2020-03-11T22:36:48.045 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:48.046 INFO:tasks.workunit.client.0.smithi059.stdout: "alerts": [], 2020-03-11T22:36:48.046 INFO:tasks.workunit.client.0.smithi059.stdout: "network_ping_times": [] 2020-03-11T22:36:48.046 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:36:48.046 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:48.046 INFO:tasks.workunit.client.0.smithi059.stdout: "pool_stats": [ 2020-03-11T22:36:48.046 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:48.047 INFO:tasks.workunit.client.0.smithi059.stdout: "poolid": 1, 2020-03-11T22:36:48.047 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pg": 1, 2020-03-11T22:36:48.047 INFO:tasks.workunit.client.0.smithi059.stdout: "stat_sum": { 2020-03-11T22:36:48.047 INFO:tasks.workunit.client.0.smithi059.stdout: "num_bytes": 48300, 2020-03-11T22:36:48.047 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects": 100, 2020-03-11T22:36:48.047 INFO:tasks.workunit.client.0.smithi059.stdout: "num_object_clones": 0, 2020-03-11T22:36:48.048 INFO:tasks.workunit.client.0.smithi059.stdout: "num_object_copies": 300, 2020-03-11T22:36:48.048 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:36:48.048 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_missing": 0, 2020-03-11T22:36:48.048 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_degraded": 100, 2020-03-11T22:36:48.048 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_misplaced": 0, 2020-03-11T22:36:48.048 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_unfound": 0, 2020-03-11T22:36:48.049 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_dirty": 100, 2020-03-11T22:36:48.049 INFO:tasks.workunit.client.0.smithi059.stdout: "num_whiteouts": 0, 2020-03-11T22:36:48.049 INFO:tasks.workunit.client.0.smithi059.stdout: "num_read": 0, 2020-03-11T22:36:48.049 INFO:tasks.workunit.client.0.smithi059.stdout: "num_read_kb": 0, 2020-03-11T22:36:48.049 INFO:tasks.workunit.client.0.smithi059.stdout: "num_write": 100, 2020-03-11T22:36:48.050 INFO:tasks.workunit.client.0.smithi059.stdout: "num_write_kb": 100, 2020-03-11T22:36:48.050 INFO:tasks.workunit.client.0.smithi059.stdout: "num_scrub_errors": 0, 2020-03-11T22:36:48.050 INFO:tasks.workunit.client.0.smithi059.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:36:48.050 INFO:tasks.workunit.client.0.smithi059.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:36:48.050 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_recovered": 0, 2020-03-11T22:36:48.050 INFO:tasks.workunit.client.0.smithi059.stdout: "num_bytes_recovered": 0, 2020-03-11T22:36:48.051 INFO:tasks.workunit.client.0.smithi059.stdout: "num_keys_recovered": 0, 2020-03-11T22:36:48.051 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_omap": 0, 2020-03-11T22:36:48.051 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:36:48.051 INFO:tasks.workunit.client.0.smithi059.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:36:48.051 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush": 0, 2020-03-11T22:36:48.051 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush_kb": 0, 2020-03-11T22:36:48.052 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict": 0, 2020-03-11T22:36:48.052 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict_kb": 0, 2020-03-11T22:36:48.052 INFO:tasks.workunit.client.0.smithi059.stdout: "num_promote": 0, 2020-03-11T22:36:48.052 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush_mode_high": 0, 2020-03-11T22:36:48.052 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush_mode_low": 0, 2020-03-11T22:36:48.052 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict_mode_some": 0, 2020-03-11T22:36:48.052 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict_mode_full": 0, 2020-03-11T22:36:48.053 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_pinned": 0, 2020-03-11T22:36:48.053 INFO:tasks.workunit.client.0.smithi059.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:36:48.053 INFO:tasks.workunit.client.0.smithi059.stdout: "num_large_omap_objects": 0, 2020-03-11T22:36:48.053 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_manifest": 0, 2020-03-11T22:36:48.053 INFO:tasks.workunit.client.0.smithi059.stdout: "num_omap_bytes": 0, 2020-03-11T22:36:48.053 INFO:tasks.workunit.client.0.smithi059.stdout: "num_omap_keys": 0, 2020-03-11T22:36:48.053 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_repaired": 0 2020-03-11T22:36:48.054 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:48.054 INFO:tasks.workunit.client.0.smithi059.stdout: "store_stats": { 2020-03-11T22:36:48.054 INFO:tasks.workunit.client.0.smithi059.stdout: "total": 0, 2020-03-11T22:36:48.054 INFO:tasks.workunit.client.0.smithi059.stdout: "available": 0, 2020-03-11T22:36:48.054 INFO:tasks.workunit.client.0.smithi059.stdout: "internally_reserved": 0, 2020-03-11T22:36:48.054 INFO:tasks.workunit.client.0.smithi059.stdout: "allocated": 19529728, 2020-03-11T22:36:48.055 INFO:tasks.workunit.client.0.smithi059.stdout: "data_stored": 143228, 2020-03-11T22:36:48.055 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed": 0, 2020-03-11T22:36:48.055 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_allocated": 0, 2020-03-11T22:36:48.055 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_original": 0, 2020-03-11T22:36:48.055 INFO:tasks.workunit.client.0.smithi059.stdout: "omap_allocated": 0, 2020-03-11T22:36:48.055 INFO:tasks.workunit.client.0.smithi059.stdout: "internal_metadata": 0 2020-03-11T22:36:48.055 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:48.056 INFO:tasks.workunit.client.0.smithi059.stdout: "log_size": 16, 2020-03-11T22:36:48.056 INFO:tasks.workunit.client.0.smithi059.stdout: "ondisk_log_size": 16, 2020-03-11T22:36:48.056 INFO:tasks.workunit.client.0.smithi059.stdout: "up": 2, 2020-03-11T22:36:48.056 INFO:tasks.workunit.client.0.smithi059.stdout: "acting": 2, 2020-03-11T22:36:48.056 INFO:tasks.workunit.client.0.smithi059.stdout: "num_store_stats": 3 2020-03-11T22:36:48.056 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:36:48.057 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:36:48.057 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_stats": [ 2020-03-11T22:36:48.057 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:48.057 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 0, 2020-03-11T22:36:48.057 INFO:tasks.workunit.client.0.smithi059.stdout: "seq": 146028888074 2020-03-11T22:36:48.057 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:48.058 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:48.058 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 1, 2020-03-11T22:36:48.058 INFO:tasks.workunit.client.0.smithi059.stdout: "seq": 42949672975 2020-03-11T22:36:48.058 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:48.058 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:48.058 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 2, 2020-03-11T22:36:48.059 INFO:tasks.workunit.client.0.smithi059.stdout: "seq": 150323855370 2020-03-11T22:36:48.059 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:36:48.059 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:36:48.059 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pg_by_state": [ 2020-03-11T22:36:48.059 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:48.059 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "active+undersized+degraded", 2020-03-11T22:36:48.060 INFO:tasks.workunit.client.0.smithi059.stdout: "num": 1 2020-03-11T22:36:48.060 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:36:48.060 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:36:48.060 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pg_by_osd": [ 2020-03-11T22:36:48.060 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:48.061 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 2, 2020-03-11T22:36:48.061 INFO:tasks.workunit.client.0.smithi059.stdout: "num_primary_pg": 0, 2020-03-11T22:36:48.061 INFO:tasks.workunit.client.0.smithi059.stdout: "num_acting_pg": 1, 2020-03-11T22:36:48.061 INFO:tasks.workunit.client.0.smithi059.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:36:48.061 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:48.061 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:48.062 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 1, 2020-03-11T22:36:48.062 INFO:tasks.workunit.client.0.smithi059.stdout: "num_primary_pg": 0, 2020-03-11T22:36:48.062 INFO:tasks.workunit.client.0.smithi059.stdout: "num_acting_pg": 0, 2020-03-11T22:36:48.062 INFO:tasks.workunit.client.0.smithi059.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:36:48.062 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:48.062 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:48.063 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 0, 2020-03-11T22:36:48.063 INFO:tasks.workunit.client.0.smithi059.stdout: "num_primary_pg": 1, 2020-03-11T22:36:48.063 INFO:tasks.workunit.client.0.smithi059.stdout: "num_acting_pg": 1, 2020-03-11T22:36:48.063 INFO:tasks.workunit.client.0.smithi059.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:36:48.063 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:36:48.063 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:36:48.064 INFO:tasks.workunit.client.0.smithi059.stdout: "purged_snaps": [ 2020-03-11T22:36:48.064 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:36:48.064 INFO:tasks.workunit.client.0.smithi059.stdout: "pool": 1, 2020-03-11T22:36:48.064 INFO:tasks.workunit.client.0.smithi059.stdout: "purged_snaps": {} 2020-03-11T22:36:48.064 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:36:48.065 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:36:48.065 INFO:tasks.workunit.client.0.smithi059.stdout: "servicemap": { 2020-03-11T22:36:48.065 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": 1, 2020-03-11T22:36:48.065 INFO:tasks.workunit.client.0.smithi059.stdout: "modified": "2020-03-11T22:35:07.878044+0000", 2020-03-11T22:36:48.065 INFO:tasks.workunit.client.0.smithi059.stdout: "services": {} 2020-03-11T22:36:48.066 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:36:48.066 INFO:tasks.workunit.client.0.smithi059.stdout: "paxos": { 2020-03-11T22:36:48.066 INFO:tasks.workunit.client.0.smithi059.stdout: "first_committed": 1, 2020-03-11T22:36:48.066 INFO:tasks.workunit.client.0.smithi059.stdout: "last_committed": 310, 2020-03-11T22:36:48.066 INFO:tasks.workunit.client.0.smithi059.stdout: "last_pn": 0, 2020-03-11T22:36:48.066 INFO:tasks.workunit.client.0.smithi059.stdout: "accepted_pn": 0 2020-03-11T22:36:48.067 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:36:48.067 INFO:tasks.workunit.client.0.smithi059.stdout:} 2020-03-11T22:36:48.068 INFO:tasks.workunit.client.0.smithi059.stderr:report 3137860655 2020-03-11T22:36:48.069 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: return 1 2020-03-11T22:36:48.069 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:528: TEST_divergent_2: expr 2 + 2 2020-03-11T22:36:48.069 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:528: TEST_divergent_2: objname=existing_4 2020-03-11T22:36:48.069 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:529: TEST_divergent_2: echo 'writing non-divergent object existing_4' 2020-03-11T22:36:48.069 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:530: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:36:48.069 INFO:tasks.workunit.client.0.smithi059.stdout:writing non-divergent object existing_4 2020-03-11T22:36:48.228 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:36:48.229 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 100 0 100 0 0 48300 0 0 16 16 active+undersized+degraded 2020-03-11T22:36:27.936281+0000 28'100 40:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:35:29.715580+0000 0'0 2020-03-11T22:35:29.715580+0000 0 2020-03-11T22:36:48.229 INFO:tasks.workunit.client.0.smithi059.stdout: 2020-03-11T22:36:48.229 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:36:48.230 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2020-03-11T22:36:48.238 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:531: TEST_divergent_2: rados -p test put existing_4 /etc/resolv.conf 2020-03-11T22:36:48.318 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:533: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:36:48.318 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:533: TEST_divergent_2: wait_for_clean 2020-03-11T22:36:48.318 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:36:48.319 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:36:48.319 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:36:48.319 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:36:48.319 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:36:48.319 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:36:48.319 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:36:48.320 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:36:48.320 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:36:48.320 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:36:48.377 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:36:48.377 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:36:48.378 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:36:48.378 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:36:48.378 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:36:48.743 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:36:48.743 INFO:tasks.workunit.client.0.smithi059.stderr:1 2020-03-11T22:36:48.743 INFO:tasks.workunit.client.0.smithi059.stderr:2' 2020-03-11T22:36:48.744 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:36:48.744 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:36:48.744 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:36:48.879 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=146028888077 2020-03-11T22:36:48.879 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-146028888077' 2020-03-11T22:36:48.879 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:36:48.880 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:36:48.975 INFO:tasks.workunit.client.0.smithi059.stderr:Error ENXIO: problem getting command descriptions from osd.1 2020-03-11T22:36:48.983 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq= 2020-03-11T22:36:48.983 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-146028888077 1-' 2020-03-11T22:36:48.983 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:36:48.984 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:36:49.119 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=150323855372 2020-03-11T22:36:49.119 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-146028888077 1- 2-150323855372' 2020-03-11T22:36:49.120 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:36:49.120 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-146028888077 2020-03-11T22:36:49.121 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:36:49.121 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:36:49.121 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-146028888077 2020-03-11T22:36:49.122 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:36:49.123 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=146028888077 2020-03-11T22:36:49.123 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 146028888077' 2020-03-11T22:36:49.123 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:36:49.123 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.0 seq 146028888077 2020-03-11T22:36:49.485 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 146028888074 -lt 146028888077 2020-03-11T22:36:49.485 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:36:50.486 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:36:50.486 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:36:50.861 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 146028888077 -lt 146028888077 2020-03-11T22:36:50.861 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:36:50.861 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1- 2020-03-11T22:36:50.862 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:36:50.862 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:36:50.863 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1- 2020-03-11T22:36:50.863 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:36:50.864 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq= 2020-03-11T22:36:50.864 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq ' 2020-03-11T22:36:50.864 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:36:50.864 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.1 seq 2020-03-11T22:36:51.228 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672975 -lt 2020-03-11T22:36:51.228 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 2079: test: 42949672975: unary operator expected 2020-03-11T22:36:51.229 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:36:51.229 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-150323855372 2020-03-11T22:36:51.229 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:36:51.230 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:36:51.230 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-150323855372 2020-03-11T22:36:51.230 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:36:51.231 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=150323855372 2020-03-11T22:36:51.231 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 150323855372' 2020-03-11T22:36:51.231 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:36:51.232 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.2 seq 150323855372 2020-03-11T22:36:51.618 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 150323855372 -lt 150323855372 2020-03-11T22:36:51.619 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:36:51.619 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:36:51.619 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:36:52.121 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:36:52.121 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:36:52.121 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:36:52.121 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:36:52.122 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:36:52.122 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:36:52.122 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:36:52.122 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:36:52.469 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:36:52.470 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:36:52.470 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:36:52.470 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:36:52.945 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:36:52.945 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:36:52.946 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:36:52.946 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:36:52.946 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:36:52.946 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:36:53.046 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:36:53.047 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:36:53.047 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:36:53.047 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:36:53.047 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:36:53.047 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:36:53.048 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:36:53.048 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:36:53.400 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:36:53.400 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:36:53.401 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:36:53.401 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:36:53.888 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:36:53.889 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:36:53.889 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:36:53.889 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:36:53.889 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:36:53.890 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:36:53.890 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:36:53.890 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:36:53.890 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:36:54.367 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:36:54.367 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:36:54.367 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 1 >= 8 )) 2020-03-11T22:36:54.368 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:36:54.368 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.2 2020-03-11T22:36:54.570 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:36:54.570 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:36:54.571 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:36:54.571 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:36:54.571 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:36:54.571 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:36:54.571 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:36:54.572 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:36:54.920 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:36:54.920 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:36:54.920 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:36:54.921 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:36:55.396 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:36:55.397 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:36:55.397 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:36:55.397 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:36:55.397 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:36:55.397 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:36:55.398 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:36:55.398 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:36:55.398 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:36:55.882 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:36:55.883 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:36:55.883 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 2 >= 8 )) 2020-03-11T22:36:55.883 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:36:55.883 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.4 2020-03-11T22:36:56.284 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:36:56.284 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:36:56.284 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:36:56.284 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:36:56.285 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:36:56.285 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:36:56.285 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:36:56.285 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:36:56.635 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:36:56.635 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:36:56.636 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:36:56.636 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:36:57.141 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:36:57.141 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:36:57.142 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:36:57.142 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:36:57.142 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:36:57.142 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:36:57.142 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:36:57.142 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:36:57.142 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:36:57.658 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:36:57.658 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:36:57.658 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 3 >= 8 )) 2020-03-11T22:36:57.659 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:36:57.659 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.8 2020-03-11T22:36:58.460 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:36:58.460 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:36:58.460 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:36:58.461 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:36:58.461 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:36:58.461 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:36:58.461 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:36:58.462 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:36:58.823 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:36:58.823 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:36:58.824 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:36:58.824 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:36:59.299 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:36:59.299 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:36:59.299 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:36:59.299 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:36:59.299 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:36:59.300 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:36:59.300 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:36:59.300 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:36:59.300 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:36:59.884 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:36:59.885 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:36:59.885 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 4 >= 8 )) 2020-03-11T22:36:59.885 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:36:59.885 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 1.6 2020-03-11T22:37:01.485 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:37:01.525 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:37:01.526 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:37:01.526 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:37:01.526 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:37:01.526 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:37:01.526 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:37:01.527 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:37:01.850 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:37:01.850 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:37:01.850 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:37:01.851 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:37:02.329 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:37:02.329 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:37:02.329 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:37:02.329 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:37:02.329 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:37:02.329 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:37:02.330 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:37:02.330 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:37:02.330 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:37:02.791 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:37:02.791 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:37:02.792 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 5 >= 8 )) 2020-03-11T22:37:02.792 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:37:02.792 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 3.2 2020-03-11T22:37:05.992 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:37:05.992 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:37:05.993 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:37:05.993 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:37:05.993 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:37:05.994 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:37:05.994 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:37:05.994 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:37:06.356 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:37:06.356 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:37:06.357 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:37:06.357 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:37:06.830 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:37:06.830 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:37:06.830 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:37:06.831 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:37:06.831 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:37:06.831 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:37:06.831 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:37:06.832 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:37:06.832 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:37:07.306 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:37:07.306 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:37:07.306 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 6 >= 8 )) 2020-03-11T22:37:07.307 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:37:07.307 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 6.4 2020-03-11T22:37:13.707 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:37:13.707 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:37:13.708 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:37:13.708 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:37:13.708 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:37:13.708 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:37:13.709 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:37:13.709 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:37:14.073 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:37:14.074 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:37:14.074 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:37:14.074 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:37:14.547 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:37:14.547 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:37:14.548 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:37:14.548 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:37:14.548 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:37:14.548 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:37:14.548 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:37:14.548 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:37:14.548 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:37:15.019 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:37:15.019 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:37:15.019 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 7 >= 8 )) 2020-03-11T22:37:15.019 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:37:15.020 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 7.3 2020-03-11T22:37:22.320 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:37:22.320 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:37:22.321 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:37:22.321 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:37:22.321 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:37:22.322 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:37:22.322 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:37:22.322 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:37:22.682 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:37:22.683 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:37:22.683 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:37:22.683 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:37:23.174 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:37:23.174 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:37:23.174 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:37:23.174 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:37:23.174 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:37:23.174 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:37:23.175 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:37:23.175 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:37:23.175 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:37:23.663 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:37:23.663 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:37:23.664 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 8 >= 8 )) 2020-03-11T22:37:23.664 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: ceph report 2020-03-11T22:37:24.136 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2020-03-11T22:37:24.137 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_fingerprint": "d95c3a98-702b-45ce-9ca1-a48c217954ec", 2020-03-11T22:37:24.137 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "15.1.0-1941-g4acc112", 2020-03-11T22:37:24.137 INFO:tasks.workunit.client.0.smithi059.stdout: "commit": "4acc1121e47039a3cdf2978ae458c10d153aa141", 2020-03-11T22:37:24.137 INFO:tasks.workunit.client.0.smithi059.stdout: "timestamp": "2020-03-11T22:37:24.133448+0000", 2020-03-11T22:37:24.137 INFO:tasks.workunit.client.0.smithi059.stdout: "tag": "", 2020-03-11T22:37:24.137 INFO:tasks.workunit.client.0.smithi059.stdout: "health": { 2020-03-11T22:37:24.138 INFO:tasks.workunit.client.0.smithi059.stdout: "status": "HEALTH_WARN", 2020-03-11T22:37:24.138 INFO:tasks.workunit.client.0.smithi059.stdout: "checks": { 2020-03-11T22:37:24.138 INFO:tasks.workunit.client.0.smithi059.stdout: "OSDMAP_FLAGS": { 2020-03-11T22:37:24.138 INFO:tasks.workunit.client.0.smithi059.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:37:24.138 INFO:tasks.workunit.client.0.smithi059.stdout: "summary": { 2020-03-11T22:37:24.138 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "nodown,noout,noin flag(s) set", 2020-03-11T22:37:24.138 INFO:tasks.workunit.client.0.smithi059.stdout: "count": 17 2020-03-11T22:37:24.139 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.139 INFO:tasks.workunit.client.0.smithi059.stdout: "detail": [], 2020-03-11T22:37:24.139 INFO:tasks.workunit.client.0.smithi059.stdout: "muted": false 2020-03-11T22:37:24.139 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.139 INFO:tasks.workunit.client.0.smithi059.stdout: "OSD_DOWN": { 2020-03-11T22:37:24.139 INFO:tasks.workunit.client.0.smithi059.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:37:24.139 INFO:tasks.workunit.client.0.smithi059.stdout: "summary": { 2020-03-11T22:37:24.139 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "1 osds down", 2020-03-11T22:37:24.140 INFO:tasks.workunit.client.0.smithi059.stdout: "count": 1 2020-03-11T22:37:24.140 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.140 INFO:tasks.workunit.client.0.smithi059.stdout: "detail": [ 2020-03-11T22:37:24.140 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.140 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "osd.1 (root=default,host=smithi059) is down" 2020-03-11T22:37:24.140 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:37:24.140 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:37:24.141 INFO:tasks.workunit.client.0.smithi059.stdout: "muted": false 2020-03-11T22:37:24.141 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.141 INFO:tasks.workunit.client.0.smithi059.stdout: "PG_DEGRADED": { 2020-03-11T22:37:24.141 INFO:tasks.workunit.client.0.smithi059.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:37:24.141 INFO:tasks.workunit.client.0.smithi059.stdout: "summary": { 2020-03-11T22:37:24.141 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded, 1 pg undersized", 2020-03-11T22:37:24.141 INFO:tasks.workunit.client.0.smithi059.stdout: "count": 2 2020-03-11T22:37:24.142 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.142 INFO:tasks.workunit.client.0.smithi059.stdout: "detail": [ 2020-03-11T22:37:24.142 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.142 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "pg 1.0 is stuck undersized for 71s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:37:24.142 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:37:24.142 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:37:24.142 INFO:tasks.workunit.client.0.smithi059.stdout: "muted": false 2020-03-11T22:37:24.143 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.143 INFO:tasks.workunit.client.0.smithi059.stdout: "POOL_APP_NOT_ENABLED": { 2020-03-11T22:37:24.143 INFO:tasks.workunit.client.0.smithi059.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:37:24.143 INFO:tasks.workunit.client.0.smithi059.stdout: "summary": { 2020-03-11T22:37:24.143 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "1 pool(s) do not have an application enabled", 2020-03-11T22:37:24.143 INFO:tasks.workunit.client.0.smithi059.stdout: "count": 1 2020-03-11T22:37:24.143 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.144 INFO:tasks.workunit.client.0.smithi059.stdout: "detail": [ 2020-03-11T22:37:24.144 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.144 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "application not enabled on pool 'test'" 2020-03-11T22:37:24.144 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.144 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.144 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2020-03-11T22:37:24.144 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:37:24.145 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:37:24.145 INFO:tasks.workunit.client.0.smithi059.stdout: "muted": false 2020-03-11T22:37:24.145 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:37:24.145 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.145 INFO:tasks.workunit.client.0.smithi059.stdout: "mutes": [] 2020-03-11T22:37:24.145 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.145 INFO:tasks.workunit.client.0.smithi059.stdout: "monmap_first_committed": 1, 2020-03-11T22:37:24.146 INFO:tasks.workunit.client.0.smithi059.stdout: "monmap_last_committed": 1, 2020-03-11T22:37:24.146 INFO:tasks.workunit.client.0.smithi059.stdout: "monmap": { 2020-03-11T22:37:24.146 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": 1, 2020-03-11T22:37:24.146 INFO:tasks.workunit.client.0.smithi059.stdout: "fsid": "cb7f2e42-1934-4145-a3c3-29bdd93f8adb", 2020-03-11T22:37:24.146 INFO:tasks.workunit.client.0.smithi059.stdout: "modified": "2020-03-11T22:35:07.115245Z", 2020-03-11T22:37:24.147 INFO:tasks.workunit.client.0.smithi059.stdout: "created": "2020-03-11T22:35:07.115245Z", 2020-03-11T22:37:24.147 INFO:tasks.workunit.client.0.smithi059.stdout: "min_mon_release": 15, 2020-03-11T22:37:24.147 INFO:tasks.workunit.client.0.smithi059.stdout: "min_mon_release_name": "octopus", 2020-03-11T22:37:24.147 INFO:tasks.workunit.client.0.smithi059.stdout: "features": { 2020-03-11T22:37:24.147 INFO:tasks.workunit.client.0.smithi059.stdout: "persistent": [ 2020-03-11T22:37:24.147 INFO:tasks.workunit.client.0.smithi059.stdout: "kraken", 2020-03-11T22:37:24.147 INFO:tasks.workunit.client.0.smithi059.stdout: "luminous", 2020-03-11T22:37:24.147 INFO:tasks.workunit.client.0.smithi059.stdout: "mimic", 2020-03-11T22:37:24.148 INFO:tasks.workunit.client.0.smithi059.stdout: "osdmap-prune", 2020-03-11T22:37:24.148 INFO:tasks.workunit.client.0.smithi059.stdout: "nautilus", 2020-03-11T22:37:24.148 INFO:tasks.workunit.client.0.smithi059.stdout: "octopus" 2020-03-11T22:37:24.148 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:37:24.148 INFO:tasks.workunit.client.0.smithi059.stdout: "optional": [] 2020-03-11T22:37:24.148 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.148 INFO:tasks.workunit.client.0.smithi059.stdout: "mons": [ 2020-03-11T22:37:24.149 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.149 INFO:tasks.workunit.client.0.smithi059.stdout: "rank": 0, 2020-03-11T22:37:24.149 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "a", 2020-03-11T22:37:24.149 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addrs": { 2020-03-11T22:37:24.149 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:37:24.149 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.149 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:37:24.149 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:7115", 2020-03-11T22:37:24.150 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 0 2020-03-11T22:37:24.150 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:37:24.150 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:37:24.150 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.150 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:7115/0", 2020-03-11T22:37:24.150 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addr": "127.0.0.1:7115/0", 2020-03-11T22:37:24.150 INFO:tasks.workunit.client.0.smithi059.stdout: "priority": 0, 2020-03-11T22:37:24.151 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 0 2020-03-11T22:37:24.151 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:37:24.151 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:37:24.151 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.151 INFO:tasks.workunit.client.0.smithi059.stdout: "quorum": [ 2020-03-11T22:37:24.151 INFO:tasks.workunit.client.0.smithi059.stdout: 0 2020-03-11T22:37:24.151 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:37:24.152 INFO:tasks.workunit.client.0.smithi059.stdout: "osdmap": { 2020-03-11T22:37:24.152 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": 40, 2020-03-11T22:37:24.152 INFO:tasks.workunit.client.0.smithi059.stdout: "fsid": "cb7f2e42-1934-4145-a3c3-29bdd93f8adb", 2020-03-11T22:37:24.152 INFO:tasks.workunit.client.0.smithi059.stdout: "created": "2020-03-11T22:35:07.877727+0000", 2020-03-11T22:37:24.152 INFO:tasks.workunit.client.0.smithi059.stdout: "modified": "2020-03-11T22:36:12.169330+0000", 2020-03-11T22:37:24.152 INFO:tasks.workunit.client.0.smithi059.stdout: "last_up_change": "2020-03-11T22:36:10.399169+0000", 2020-03-11T22:37:24.152 INFO:tasks.workunit.client.0.smithi059.stdout: "last_in_change": "2020-03-11T22:36:11.901843+0000", 2020-03-11T22:37:24.152 INFO:tasks.workunit.client.0.smithi059.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-03-11T22:37:24.153 INFO:tasks.workunit.client.0.smithi059.stdout: "flags_num": 5800384, 2020-03-11T22:37:24.153 INFO:tasks.workunit.client.0.smithi059.stdout: "flags_set": [ 2020-03-11T22:37:24.153 INFO:tasks.workunit.client.0.smithi059.stdout: "nodown", 2020-03-11T22:37:24.153 INFO:tasks.workunit.client.0.smithi059.stdout: "noin", 2020-03-11T22:37:24.153 INFO:tasks.workunit.client.0.smithi059.stdout: "noout", 2020-03-11T22:37:24.153 INFO:tasks.workunit.client.0.smithi059.stdout: "pglog_hardlimit", 2020-03-11T22:37:24.154 INFO:tasks.workunit.client.0.smithi059.stdout: "purged_snapdirs", 2020-03-11T22:37:24.154 INFO:tasks.workunit.client.0.smithi059.stdout: "recovery_deletes", 2020-03-11T22:37:24.154 INFO:tasks.workunit.client.0.smithi059.stdout: "sortbitwise" 2020-03-11T22:37:24.154 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:37:24.154 INFO:tasks.workunit.client.0.smithi059.stdout: "crush_version": 7, 2020-03-11T22:37:24.154 INFO:tasks.workunit.client.0.smithi059.stdout: "full_ratio": 0.99000000953674316, 2020-03-11T22:37:24.154 INFO:tasks.workunit.client.0.smithi059.stdout: "backfillfull_ratio": 0.99000000953674316, 2020-03-11T22:37:24.155 INFO:tasks.workunit.client.0.smithi059.stdout: "nearfull_ratio": 0.85000002384185791, 2020-03-11T22:37:24.155 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_snapshot": "", 2020-03-11T22:37:24.155 INFO:tasks.workunit.client.0.smithi059.stdout: "pool_max": 1, 2020-03-11T22:37:24.155 INFO:tasks.workunit.client.0.smithi059.stdout: "max_osd": 3, 2020-03-11T22:37:24.155 INFO:tasks.workunit.client.0.smithi059.stdout: "require_min_compat_client": "jewel", 2020-03-11T22:37:24.155 INFO:tasks.workunit.client.0.smithi059.stdout: "min_compat_client": "jewel", 2020-03-11T22:37:24.155 INFO:tasks.workunit.client.0.smithi059.stdout: "require_osd_release": "octopus", 2020-03-11T22:37:24.155 INFO:tasks.workunit.client.0.smithi059.stdout: "pools": [ 2020-03-11T22:37:24.156 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.156 INFO:tasks.workunit.client.0.smithi059.stdout: "pool": 1, 2020-03-11T22:37:24.156 INFO:tasks.workunit.client.0.smithi059.stdout: "pool_name": "test", 2020-03-11T22:37:24.156 INFO:tasks.workunit.client.0.smithi059.stdout: "create_time": "2020-03-11T22:35:29.665394+0000", 2020-03-11T22:37:24.156 INFO:tasks.workunit.client.0.smithi059.stdout: "flags": 1, 2020-03-11T22:37:24.156 INFO:tasks.workunit.client.0.smithi059.stdout: "flags_names": "hashpspool", 2020-03-11T22:37:24.156 INFO:tasks.workunit.client.0.smithi059.stdout: "type": 1, 2020-03-11T22:37:24.157 INFO:tasks.workunit.client.0.smithi059.stdout: "size": 3, 2020-03-11T22:37:24.157 INFO:tasks.workunit.client.0.smithi059.stdout: "min_size": 2, 2020-03-11T22:37:24.157 INFO:tasks.workunit.client.0.smithi059.stdout: "crush_rule": 0, 2020-03-11T22:37:24.157 INFO:tasks.workunit.client.0.smithi059.stdout: "object_hash": 2, 2020-03-11T22:37:24.157 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_autoscale_mode": "off", 2020-03-11T22:37:24.157 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_num": 1, 2020-03-11T22:37:24.157 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_placement_num": 1, 2020-03-11T22:37:24.158 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_placement_num_target": 1, 2020-03-11T22:37:24.158 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_num_target": 1, 2020-03-11T22:37:24.158 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_num_pending": 1, 2020-03-11T22:37:24.158 INFO:tasks.workunit.client.0.smithi059.stdout: "last_pg_merge_meta": { 2020-03-11T22:37:24.158 INFO:tasks.workunit.client.0.smithi059.stdout: "source_pgid": "0.0", 2020-03-11T22:37:24.158 INFO:tasks.workunit.client.0.smithi059.stdout: "ready_epoch": 0, 2020-03-11T22:37:24.159 INFO:tasks.workunit.client.0.smithi059.stdout: "last_epoch_started": 0, 2020-03-11T22:37:24.159 INFO:tasks.workunit.client.0.smithi059.stdout: "last_epoch_clean": 0, 2020-03-11T22:37:24.159 INFO:tasks.workunit.client.0.smithi059.stdout: "source_version": "0'0", 2020-03-11T22:37:24.159 INFO:tasks.workunit.client.0.smithi059.stdout: "target_version": "0'0" 2020-03-11T22:37:24.159 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.159 INFO:tasks.workunit.client.0.smithi059.stdout: "last_change": "28", 2020-03-11T22:37:24.159 INFO:tasks.workunit.client.0.smithi059.stdout: "last_force_op_resend": "0", 2020-03-11T22:37:24.160 INFO:tasks.workunit.client.0.smithi059.stdout: "last_force_op_resend_prenautilus": "0", 2020-03-11T22:37:24.160 INFO:tasks.workunit.client.0.smithi059.stdout: "last_force_op_resend_preluminous": "0", 2020-03-11T22:37:24.160 INFO:tasks.workunit.client.0.smithi059.stdout: "auid": 0, 2020-03-11T22:37:24.160 INFO:tasks.workunit.client.0.smithi059.stdout: "snap_mode": "selfmanaged", 2020-03-11T22:37:24.160 INFO:tasks.workunit.client.0.smithi059.stdout: "snap_seq": 0, 2020-03-11T22:37:24.160 INFO:tasks.workunit.client.0.smithi059.stdout: "snap_epoch": 0, 2020-03-11T22:37:24.160 INFO:tasks.workunit.client.0.smithi059.stdout: "pool_snaps": [], 2020-03-11T22:37:24.161 INFO:tasks.workunit.client.0.smithi059.stdout: "removed_snaps": "[]", 2020-03-11T22:37:24.161 INFO:tasks.workunit.client.0.smithi059.stdout: "quota_max_bytes": 0, 2020-03-11T22:37:24.161 INFO:tasks.workunit.client.0.smithi059.stdout: "quota_max_objects": 0, 2020-03-11T22:37:24.161 INFO:tasks.workunit.client.0.smithi059.stdout: "tiers": [], 2020-03-11T22:37:24.161 INFO:tasks.workunit.client.0.smithi059.stdout: "tier_of": -1, 2020-03-11T22:37:24.161 INFO:tasks.workunit.client.0.smithi059.stdout: "read_tier": -1, 2020-03-11T22:37:24.161 INFO:tasks.workunit.client.0.smithi059.stdout: "write_tier": -1, 2020-03-11T22:37:24.162 INFO:tasks.workunit.client.0.smithi059.stdout: "cache_mode": "none", 2020-03-11T22:37:24.162 INFO:tasks.workunit.client.0.smithi059.stdout: "target_max_bytes": 0, 2020-03-11T22:37:24.162 INFO:tasks.workunit.client.0.smithi059.stdout: "target_max_objects": 0, 2020-03-11T22:37:24.162 INFO:tasks.workunit.client.0.smithi059.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-03-11T22:37:24.162 INFO:tasks.workunit.client.0.smithi059.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-03-11T22:37:24.162 INFO:tasks.workunit.client.0.smithi059.stdout: "cache_target_full_ratio_micro": 800000, 2020-03-11T22:37:24.163 INFO:tasks.workunit.client.0.smithi059.stdout: "cache_min_flush_age": 0, 2020-03-11T22:37:24.163 INFO:tasks.workunit.client.0.smithi059.stdout: "cache_min_evict_age": 0, 2020-03-11T22:37:24.163 INFO:tasks.workunit.client.0.smithi059.stdout: "erasure_code_profile": "", 2020-03-11T22:37:24.163 INFO:tasks.workunit.client.0.smithi059.stdout: "hit_set_params": { 2020-03-11T22:37:24.163 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "none" 2020-03-11T22:37:24.163 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.163 INFO:tasks.workunit.client.0.smithi059.stdout: "hit_set_period": 0, 2020-03-11T22:37:24.163 INFO:tasks.workunit.client.0.smithi059.stdout: "hit_set_count": 0, 2020-03-11T22:37:24.164 INFO:tasks.workunit.client.0.smithi059.stdout: "use_gmt_hitset": true, 2020-03-11T22:37:24.164 INFO:tasks.workunit.client.0.smithi059.stdout: "min_read_recency_for_promote": 0, 2020-03-11T22:37:24.164 INFO:tasks.workunit.client.0.smithi059.stdout: "min_write_recency_for_promote": 0, 2020-03-11T22:37:24.164 INFO:tasks.workunit.client.0.smithi059.stdout: "hit_set_grade_decay_rate": 0, 2020-03-11T22:37:24.164 INFO:tasks.workunit.client.0.smithi059.stdout: "hit_set_search_last_n": 0, 2020-03-11T22:37:24.164 INFO:tasks.workunit.client.0.smithi059.stdout: "grade_table": [], 2020-03-11T22:37:24.164 INFO:tasks.workunit.client.0.smithi059.stdout: "stripe_width": 0, 2020-03-11T22:37:24.165 INFO:tasks.workunit.client.0.smithi059.stdout: "expected_num_objects": 0, 2020-03-11T22:37:24.165 INFO:tasks.workunit.client.0.smithi059.stdout: "fast_read": false, 2020-03-11T22:37:24.165 INFO:tasks.workunit.client.0.smithi059.stdout: "options": {}, 2020-03-11T22:37:24.165 INFO:tasks.workunit.client.0.smithi059.stdout: "application_metadata": {} 2020-03-11T22:37:24.165 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:37:24.165 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:37:24.165 INFO:tasks.workunit.client.0.smithi059.stdout: "osds": [ 2020-03-11T22:37:24.166 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.166 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 0, 2020-03-11T22:37:24.166 INFO:tasks.workunit.client.0.smithi059.stdout: "uuid": "6b6c1a2d-a55b-410f-9b5c-d5f724d8ff3d", 2020-03-11T22:37:24.166 INFO:tasks.workunit.client.0.smithi059.stdout: "up": 1, 2020-03-11T22:37:24.166 INFO:tasks.workunit.client.0.smithi059.stdout: "in": 1, 2020-03-11T22:37:24.166 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 1, 2020-03-11T22:37:24.166 INFO:tasks.workunit.client.0.smithi059.stdout: "primary_affinity": 1, 2020-03-11T22:37:24.167 INFO:tasks.workunit.client.0.smithi059.stdout: "last_clean_begin": 5, 2020-03-11T22:37:24.167 INFO:tasks.workunit.client.0.smithi059.stdout: "last_clean_end": 28, 2020-03-11T22:37:24.167 INFO:tasks.workunit.client.0.smithi059.stdout: "up_from": 34, 2020-03-11T22:37:24.167 INFO:tasks.workunit.client.0.smithi059.stdout: "up_thru": 39, 2020-03-11T22:37:24.167 INFO:tasks.workunit.client.0.smithi059.stdout: "down_at": 29, 2020-03-11T22:37:24.167 INFO:tasks.workunit.client.0.smithi059.stdout: "lost_at": 0, 2020-03-11T22:37:24.167 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addrs": { 2020-03-11T22:37:24.168 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:37:24.168 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.168 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:37:24.168 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6802", 2020-03-11T22:37:24.168 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 30799 2020-03-11T22:37:24.168 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.168 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.168 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:37:24.169 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6803", 2020-03-11T22:37:24.169 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 30799 2020-03-11T22:37:24.169 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:37:24.169 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:37:24.169 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.169 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_addrs": { 2020-03-11T22:37:24.169 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:37:24.170 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.170 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:37:24.170 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6804", 2020-03-11T22:37:24.170 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 30799 2020-03-11T22:37:24.170 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.170 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.170 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:37:24.171 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6805", 2020-03-11T22:37:24.171 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 30799 2020-03-11T22:37:24.171 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:37:24.171 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:37:24.171 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.171 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_back_addrs": { 2020-03-11T22:37:24.172 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:37:24.172 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.172 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:37:24.172 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6808", 2020-03-11T22:37:24.172 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 30799 2020-03-11T22:37:24.172 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.172 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.173 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:37:24.173 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6809", 2020-03-11T22:37:24.173 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 30799 2020-03-11T22:37:24.173 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:37:24.173 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:37:24.173 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.174 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_front_addrs": { 2020-03-11T22:37:24.174 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:37:24.174 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.174 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:37:24.174 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6806", 2020-03-11T22:37:24.174 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 30799 2020-03-11T22:37:24.174 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.174 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.175 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:37:24.175 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6807", 2020-03-11T22:37:24.175 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 30799 2020-03-11T22:37:24.175 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:37:24.175 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:37:24.175 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.175 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addr": "127.0.0.1:6803/30799", 2020-03-11T22:37:24.176 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_addr": "127.0.0.1:6805/30799", 2020-03-11T22:37:24.176 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_back_addr": "127.0.0.1:6809/30799", 2020-03-11T22:37:24.176 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_front_addr": "127.0.0.1:6807/30799", 2020-03-11T22:37:24.176 INFO:tasks.workunit.client.0.smithi059.stdout: "state": [ 2020-03-11T22:37:24.176 INFO:tasks.workunit.client.0.smithi059.stdout: "exists", 2020-03-11T22:37:24.176 INFO:tasks.workunit.client.0.smithi059.stdout: "up" 2020-03-11T22:37:24.176 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:37:24.177 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.177 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.177 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 1, 2020-03-11T22:37:24.177 INFO:tasks.workunit.client.0.smithi059.stdout: "uuid": "55fb3134-fe56-4d99-9135-8d0c47f417da", 2020-03-11T22:37:24.177 INFO:tasks.workunit.client.0.smithi059.stdout: "up": 0, 2020-03-11T22:37:24.177 INFO:tasks.workunit.client.0.smithi059.stdout: "in": 1, 2020-03-11T22:37:24.177 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 1, 2020-03-11T22:37:24.178 INFO:tasks.workunit.client.0.smithi059.stdout: "primary_affinity": 1, 2020-03-11T22:37:24.178 INFO:tasks.workunit.client.0.smithi059.stdout: "last_clean_begin": 0, 2020-03-11T22:37:24.178 INFO:tasks.workunit.client.0.smithi059.stdout: "last_clean_end": 0, 2020-03-11T22:37:24.178 INFO:tasks.workunit.client.0.smithi059.stdout: "up_from": 10, 2020-03-11T22:37:24.178 INFO:tasks.workunit.client.0.smithi059.stdout: "up_thru": 22, 2020-03-11T22:37:24.178 INFO:tasks.workunit.client.0.smithi059.stdout: "down_at": 30, 2020-03-11T22:37:24.178 INFO:tasks.workunit.client.0.smithi059.stdout: "lost_at": 0, 2020-03-11T22:37:24.178 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addrs": { 2020-03-11T22:37:24.179 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:37:24.179 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.179 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:37:24.179 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:37:24.179 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 26394 2020-03-11T22:37:24.179 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.179 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.180 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:37:24.180 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:37:24.180 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 26394 2020-03-11T22:37:24.180 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:37:24.180 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:37:24.180 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.180 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_addrs": { 2020-03-11T22:37:24.180 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:37:24.181 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.181 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:37:24.181 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:37:24.181 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 26394 2020-03-11T22:37:24.181 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.181 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.181 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:37:24.182 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:37:24.182 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 26394 2020-03-11T22:37:24.182 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:37:24.182 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:37:24.182 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.182 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_back_addrs": { 2020-03-11T22:37:24.182 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:37:24.182 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.183 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:37:24.183 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:37:24.183 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 26394 2020-03-11T22:37:24.183 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.183 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.183 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:37:24.184 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:37:24.184 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 26394 2020-03-11T22:37:24.184 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:37:24.184 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:37:24.184 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.184 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_front_addrs": { 2020-03-11T22:37:24.184 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:37:24.184 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.185 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:37:24.185 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:37:24.185 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 26394 2020-03-11T22:37:24.185 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.185 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.185 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:37:24.185 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:37:24.186 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 26394 2020-03-11T22:37:24.186 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:37:24.186 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:37:24.186 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.186 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addr": "127.0.0.1:6811/26394", 2020-03-11T22:37:24.186 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_addr": "127.0.0.1:6813/26394", 2020-03-11T22:37:24.186 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_back_addr": "127.0.0.1:6817/26394", 2020-03-11T22:37:24.187 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_front_addr": "127.0.0.1:6815/26394", 2020-03-11T22:37:24.187 INFO:tasks.workunit.client.0.smithi059.stdout: "state": [ 2020-03-11T22:37:24.187 INFO:tasks.workunit.client.0.smithi059.stdout: "exists" 2020-03-11T22:37:24.187 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:37:24.187 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.187 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.187 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 2, 2020-03-11T22:37:24.187 INFO:tasks.workunit.client.0.smithi059.stdout: "uuid": "3aa1b594-c731-4355-b2e6-05a8c3d7bb4b", 2020-03-11T22:37:24.188 INFO:tasks.workunit.client.0.smithi059.stdout: "up": 1, 2020-03-11T22:37:24.188 INFO:tasks.workunit.client.0.smithi059.stdout: "in": 1, 2020-03-11T22:37:24.188 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 1, 2020-03-11T22:37:24.188 INFO:tasks.workunit.client.0.smithi059.stdout: "primary_affinity": 1, 2020-03-11T22:37:24.188 INFO:tasks.workunit.client.0.smithi059.stdout: "last_clean_begin": 15, 2020-03-11T22:37:24.188 INFO:tasks.workunit.client.0.smithi059.stdout: "last_clean_end": 28, 2020-03-11T22:37:24.188 INFO:tasks.workunit.client.0.smithi059.stdout: "up_from": 35, 2020-03-11T22:37:24.189 INFO:tasks.workunit.client.0.smithi059.stdout: "up_thru": 0, 2020-03-11T22:37:24.189 INFO:tasks.workunit.client.0.smithi059.stdout: "down_at": 31, 2020-03-11T22:37:24.189 INFO:tasks.workunit.client.0.smithi059.stdout: "lost_at": 0, 2020-03-11T22:37:24.189 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addrs": { 2020-03-11T22:37:24.189 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:37:24.189 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.189 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:37:24.190 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:37:24.190 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 30993 2020-03-11T22:37:24.190 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.190 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.190 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:37:24.190 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:37:24.190 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 30993 2020-03-11T22:37:24.190 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:37:24.191 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:37:24.191 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.191 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_addrs": { 2020-03-11T22:37:24.191 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:37:24.191 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.191 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:37:24.191 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:37:24.191 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 30993 2020-03-11T22:37:24.192 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.192 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.192 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:37:24.192 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:37:24.192 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 30993 2020-03-11T22:37:24.192 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:37:24.192 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:37:24.193 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.193 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_back_addrs": { 2020-03-11T22:37:24.193 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:37:24.193 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.193 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:37:24.193 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:37:24.193 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 30993 2020-03-11T22:37:24.193 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.194 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.194 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:37:24.194 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:37:24.194 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 30993 2020-03-11T22:37:24.194 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:37:24.194 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:37:24.194 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.195 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_front_addrs": { 2020-03-11T22:37:24.196 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:37:24.197 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.197 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:37:24.197 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:37:24.197 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 30993 2020-03-11T22:37:24.197 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.197 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.197 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:37:24.197 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:37:24.198 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 30993 2020-03-11T22:37:24.198 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:37:24.198 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:37:24.198 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.198 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addr": "127.0.0.1:6811/30993", 2020-03-11T22:37:24.198 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_addr": "127.0.0.1:6813/30993", 2020-03-11T22:37:24.198 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_back_addr": "127.0.0.1:6817/30993", 2020-03-11T22:37:24.199 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_front_addr": "127.0.0.1:6815/30993", 2020-03-11T22:37:24.199 INFO:tasks.workunit.client.0.smithi059.stdout: "state": [ 2020-03-11T22:37:24.199 INFO:tasks.workunit.client.0.smithi059.stdout: "exists", 2020-03-11T22:37:24.199 INFO:tasks.workunit.client.0.smithi059.stdout: "up" 2020-03-11T22:37:24.199 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:37:24.199 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:37:24.199 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:37:24.199 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_xinfo": [ 2020-03-11T22:37:24.200 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.200 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 0, 2020-03-11T22:37:24.200 INFO:tasks.workunit.client.0.smithi059.stdout: "down_stamp": "2020-03-11T22:36:03.766729+0000", 2020-03-11T22:37:24.200 INFO:tasks.workunit.client.0.smithi059.stdout: "laggy_probability": 0, 2020-03-11T22:37:24.200 INFO:tasks.workunit.client.0.smithi059.stdout: "laggy_interval": 0, 2020-03-11T22:37:24.200 INFO:tasks.workunit.client.0.smithi059.stdout: "features": 4540138292836696063, 2020-03-11T22:37:24.200 INFO:tasks.workunit.client.0.smithi059.stdout: "old_weight": 0, 2020-03-11T22:37:24.201 INFO:tasks.workunit.client.0.smithi059.stdout: "last_purged_snaps_scrub": "2020-03-11T22:35:14.607103+0000", 2020-03-11T22:37:24.201 INFO:tasks.workunit.client.0.smithi059.stdout: "dead_epoch": 0 2020-03-11T22:37:24.201 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.201 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.201 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 1, 2020-03-11T22:37:24.201 INFO:tasks.workunit.client.0.smithi059.stdout: "down_stamp": "2020-03-11T22:36:04.292275+0000", 2020-03-11T22:37:24.201 INFO:tasks.workunit.client.0.smithi059.stdout: "laggy_probability": 0, 2020-03-11T22:37:24.201 INFO:tasks.workunit.client.0.smithi059.stdout: "laggy_interval": 0, 2020-03-11T22:37:24.202 INFO:tasks.workunit.client.0.smithi059.stdout: "features": 4540138292836696063, 2020-03-11T22:37:24.202 INFO:tasks.workunit.client.0.smithi059.stdout: "old_weight": 0, 2020-03-11T22:37:24.202 INFO:tasks.workunit.client.0.smithi059.stdout: "last_purged_snaps_scrub": "2020-03-11T22:35:20.928320+0000", 2020-03-11T22:37:24.202 INFO:tasks.workunit.client.0.smithi059.stdout: "dead_epoch": 0 2020-03-11T22:37:24.202 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.202 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.202 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 2, 2020-03-11T22:37:24.203 INFO:tasks.workunit.client.0.smithi059.stdout: "down_stamp": "2020-03-11T22:36:04.841346+0000", 2020-03-11T22:37:24.203 INFO:tasks.workunit.client.0.smithi059.stdout: "laggy_probability": 0, 2020-03-11T22:37:24.203 INFO:tasks.workunit.client.0.smithi059.stdout: "laggy_interval": 0, 2020-03-11T22:37:24.203 INFO:tasks.workunit.client.0.smithi059.stdout: "features": 4540138292836696063, 2020-03-11T22:37:24.203 INFO:tasks.workunit.client.0.smithi059.stdout: "old_weight": 0, 2020-03-11T22:37:24.203 INFO:tasks.workunit.client.0.smithi059.stdout: "last_purged_snaps_scrub": "2020-03-11T22:35:27.361869+0000", 2020-03-11T22:37:24.203 INFO:tasks.workunit.client.0.smithi059.stdout: "dead_epoch": 0 2020-03-11T22:37:24.203 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:37:24.204 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:37:24.204 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_upmap": [], 2020-03-11T22:37:24.204 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_upmap_items": [], 2020-03-11T22:37:24.204 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_temp": [], 2020-03-11T22:37:24.204 INFO:tasks.workunit.client.0.smithi059.stdout: "primary_temp": [], 2020-03-11T22:37:24.204 INFO:tasks.workunit.client.0.smithi059.stdout: "blacklist": {}, 2020-03-11T22:37:24.204 INFO:tasks.workunit.client.0.smithi059.stdout: "erasure_code_profiles": { 2020-03-11T22:37:24.205 INFO:tasks.workunit.client.0.smithi059.stdout: "default": { 2020-03-11T22:37:24.205 INFO:tasks.workunit.client.0.smithi059.stdout: "k": "2", 2020-03-11T22:37:24.205 INFO:tasks.workunit.client.0.smithi059.stdout: "m": "2", 2020-03-11T22:37:24.205 INFO:tasks.workunit.client.0.smithi059.stdout: "plugin": "jerasure", 2020-03-11T22:37:24.205 INFO:tasks.workunit.client.0.smithi059.stdout: "technique": "reed_sol_van" 2020-03-11T22:37:24.205 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:37:24.205 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.205 INFO:tasks.workunit.client.0.smithi059.stdout: "removed_snaps_queue": [], 2020-03-11T22:37:24.206 INFO:tasks.workunit.client.0.smithi059.stdout: "new_removed_snaps": [], 2020-03-11T22:37:24.206 INFO:tasks.workunit.client.0.smithi059.stdout: "new_purged_snaps": [], 2020-03-11T22:37:24.206 INFO:tasks.workunit.client.0.smithi059.stdout: "crush_node_flags": {}, 2020-03-11T22:37:24.206 INFO:tasks.workunit.client.0.smithi059.stdout: "device_class_flags": {} 2020-03-11T22:37:24.206 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.206 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_metadata": [ 2020-03-11T22:37:24.206 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.207 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 0, 2020-03-11T22:37:24.207 INFO:tasks.workunit.client.0.smithi059.stdout: "arch": "x86_64", 2020-03-11T22:37:24.207 INFO:tasks.workunit.client.0.smithi059.stdout: "back_addr": "[v2:127.0.0.1:6804/30799,v1:127.0.0.1:6805/30799]", 2020-03-11T22:37:24.207 INFO:tasks.workunit.client.0.smithi059.stdout: "back_iface": "", 2020-03-11T22:37:24.207 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs": "1", 2020-03-11T22:37:24.207 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:37:24.207 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:37:24.208 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:37:24.208 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:37:24.208 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:37:24.208 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:37:24.208 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:37:24.208 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2020-03-11T22:37:24.208 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:37:24.208 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:37:24.209 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:37:24.209 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:37:24.209 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_release": "octopus", 2020-03-11T22:37:24.209 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:37:24.209 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:37:24.209 INFO:tasks.workunit.client.0.smithi059.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-03-11T22:37:24.209 INFO:tasks.workunit.client.0.smithi059.stdout: "default_device_class": "hdd", 2020-03-11T22:37:24.210 INFO:tasks.workunit.client.0.smithi059.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4J2P7", 2020-03-11T22:37:24.210 INFO:tasks.workunit.client.0.smithi059.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:37:24.210 INFO:tasks.workunit.client.0.smithi059.stdout: "devices": "sda", 2020-03-11T22:37:24.210 INFO:tasks.workunit.client.0.smithi059.stdout: "distro": "centos", 2020-03-11T22:37:24.210 INFO:tasks.workunit.client.0.smithi059.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:37:24.210 INFO:tasks.workunit.client.0.smithi059.stdout: "distro_version": "8", 2020-03-11T22:37:24.210 INFO:tasks.workunit.client.0.smithi059.stdout: "front_addr": "[v2:127.0.0.1:6802/30799,v1:127.0.0.1:6803/30799]", 2020-03-11T22:37:24.211 INFO:tasks.workunit.client.0.smithi059.stdout: "front_iface": "", 2020-03-11T22:37:24.211 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/30799,v1:127.0.0.1:6809/30799]", 2020-03-11T22:37:24.211 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/30799,v1:127.0.0.1:6807/30799]", 2020-03-11T22:37:24.211 INFO:tasks.workunit.client.0.smithi059.stdout: "hostname": "smithi059", 2020-03-11T22:37:24.211 INFO:tasks.workunit.client.0.smithi059.stdout: "journal_rotational": "1", 2020-03-11T22:37:24.211 INFO:tasks.workunit.client.0.smithi059.stdout: "kernel_description": "#1 SMP Wed Feb 5 02:00:39 UTC 2020", 2020-03-11T22:37:24.212 INFO:tasks.workunit.client.0.smithi059.stdout: "kernel_version": "4.18.0-147.5.1.el8_1.x86_64", 2020-03-11T22:37:24.212 INFO:tasks.workunit.client.0.smithi059.stdout: "mem_swap_kb": "0", 2020-03-11T22:37:24.212 INFO:tasks.workunit.client.0.smithi059.stdout: "mem_total_kb": "32662084", 2020-03-11T22:37:24.212 INFO:tasks.workunit.client.0.smithi059.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:37:24.212 INFO:tasks.workunit.client.0.smithi059.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:37:24.212 INFO:tasks.workunit.client.0.smithi059.stdout: "os": "Linux", 2020-03-11T22:37:24.212 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_data": "td/divergent-priors/0", 2020-03-11T22:37:24.212 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:37:24.213 INFO:tasks.workunit.client.0.smithi059.stdout: "rotational": "1" 2020-03-11T22:37:24.213 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.213 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.213 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 1, 2020-03-11T22:37:24.213 INFO:tasks.workunit.client.0.smithi059.stdout: "arch": "x86_64", 2020-03-11T22:37:24.213 INFO:tasks.workunit.client.0.smithi059.stdout: "back_addr": "[v2:127.0.0.1:6812/26394,v1:127.0.0.1:6813/26394]", 2020-03-11T22:37:24.213 INFO:tasks.workunit.client.0.smithi059.stdout: "back_iface": "", 2020-03-11T22:37:24.214 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs": "1", 2020-03-11T22:37:24.214 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:37:24.214 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:37:24.214 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:37:24.214 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:37:24.214 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:37:24.214 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:37:24.215 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:37:24.215 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2020-03-11T22:37:24.215 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:37:24.215 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:37:24.215 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:37:24.215 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:37:24.215 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_release": "octopus", 2020-03-11T22:37:24.216 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:37:24.216 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:37:24.216 INFO:tasks.workunit.client.0.smithi059.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-03-11T22:37:24.216 INFO:tasks.workunit.client.0.smithi059.stdout: "default_device_class": "hdd", 2020-03-11T22:37:24.216 INFO:tasks.workunit.client.0.smithi059.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4J2P7", 2020-03-11T22:37:24.216 INFO:tasks.workunit.client.0.smithi059.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:37:24.217 INFO:tasks.workunit.client.0.smithi059.stdout: "devices": "sda", 2020-03-11T22:37:24.217 INFO:tasks.workunit.client.0.smithi059.stdout: "distro": "centos", 2020-03-11T22:37:24.217 INFO:tasks.workunit.client.0.smithi059.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:37:24.217 INFO:tasks.workunit.client.0.smithi059.stdout: "distro_version": "8", 2020-03-11T22:37:24.217 INFO:tasks.workunit.client.0.smithi059.stdout: "front_addr": "[v2:127.0.0.1:6810/26394,v1:127.0.0.1:6811/26394]", 2020-03-11T22:37:24.217 INFO:tasks.workunit.client.0.smithi059.stdout: "front_iface": "", 2020-03-11T22:37:24.217 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/26394,v1:127.0.0.1:6817/26394]", 2020-03-11T22:37:24.218 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/26394,v1:127.0.0.1:6815/26394]", 2020-03-11T22:37:24.218 INFO:tasks.workunit.client.0.smithi059.stdout: "hostname": "smithi059", 2020-03-11T22:37:24.218 INFO:tasks.workunit.client.0.smithi059.stdout: "journal_rotational": "1", 2020-03-11T22:37:24.218 INFO:tasks.workunit.client.0.smithi059.stdout: "kernel_description": "#1 SMP Wed Feb 5 02:00:39 UTC 2020", 2020-03-11T22:37:24.218 INFO:tasks.workunit.client.0.smithi059.stdout: "kernel_version": "4.18.0-147.5.1.el8_1.x86_64", 2020-03-11T22:37:24.218 INFO:tasks.workunit.client.0.smithi059.stdout: "mem_swap_kb": "0", 2020-03-11T22:37:24.218 INFO:tasks.workunit.client.0.smithi059.stdout: "mem_total_kb": "32662084", 2020-03-11T22:37:24.218 INFO:tasks.workunit.client.0.smithi059.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:37:24.219 INFO:tasks.workunit.client.0.smithi059.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:37:24.219 INFO:tasks.workunit.client.0.smithi059.stdout: "os": "Linux", 2020-03-11T22:37:24.219 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_data": "td/divergent-priors/1", 2020-03-11T22:37:24.219 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:37:24.219 INFO:tasks.workunit.client.0.smithi059.stdout: "rotational": "1" 2020-03-11T22:37:24.219 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.219 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.220 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 2, 2020-03-11T22:37:24.220 INFO:tasks.workunit.client.0.smithi059.stdout: "arch": "x86_64", 2020-03-11T22:37:24.220 INFO:tasks.workunit.client.0.smithi059.stdout: "back_addr": "[v2:127.0.0.1:6812/30993,v1:127.0.0.1:6813/30993]", 2020-03-11T22:37:24.220 INFO:tasks.workunit.client.0.smithi059.stdout: "back_iface": "", 2020-03-11T22:37:24.220 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs": "1", 2020-03-11T22:37:24.220 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:37:24.220 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:37:24.220 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:37:24.221 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:37:24.221 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:37:24.221 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:37:24.221 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:37:24.221 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2020-03-11T22:37:24.221 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:37:24.221 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:37:24.222 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:37:24.222 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:37:24.222 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_release": "octopus", 2020-03-11T22:37:24.222 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:37:24.222 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:37:24.222 INFO:tasks.workunit.client.0.smithi059.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-03-11T22:37:24.222 INFO:tasks.workunit.client.0.smithi059.stdout: "default_device_class": "hdd", 2020-03-11T22:37:24.222 INFO:tasks.workunit.client.0.smithi059.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4J2P7", 2020-03-11T22:37:24.223 INFO:tasks.workunit.client.0.smithi059.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:37:24.223 INFO:tasks.workunit.client.0.smithi059.stdout: "devices": "sda", 2020-03-11T22:37:24.223 INFO:tasks.workunit.client.0.smithi059.stdout: "distro": "centos", 2020-03-11T22:37:24.223 INFO:tasks.workunit.client.0.smithi059.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:37:24.223 INFO:tasks.workunit.client.0.smithi059.stdout: "distro_version": "8", 2020-03-11T22:37:24.223 INFO:tasks.workunit.client.0.smithi059.stdout: "front_addr": "[v2:127.0.0.1:6810/30993,v1:127.0.0.1:6811/30993]", 2020-03-11T22:37:24.223 INFO:tasks.workunit.client.0.smithi059.stdout: "front_iface": "", 2020-03-11T22:37:24.223 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/30993,v1:127.0.0.1:6817/30993]", 2020-03-11T22:37:24.224 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/30993,v1:127.0.0.1:6815/30993]", 2020-03-11T22:37:24.224 INFO:tasks.workunit.client.0.smithi059.stdout: "hostname": "smithi059", 2020-03-11T22:37:24.224 INFO:tasks.workunit.client.0.smithi059.stdout: "journal_rotational": "1", 2020-03-11T22:37:24.224 INFO:tasks.workunit.client.0.smithi059.stdout: "kernel_description": "#1 SMP Wed Feb 5 02:00:39 UTC 2020", 2020-03-11T22:37:24.224 INFO:tasks.workunit.client.0.smithi059.stdout: "kernel_version": "4.18.0-147.5.1.el8_1.x86_64", 2020-03-11T22:37:24.224 INFO:tasks.workunit.client.0.smithi059.stdout: "mem_swap_kb": "0", 2020-03-11T22:37:24.224 INFO:tasks.workunit.client.0.smithi059.stdout: "mem_total_kb": "32662084", 2020-03-11T22:37:24.225 INFO:tasks.workunit.client.0.smithi059.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:37:24.225 INFO:tasks.workunit.client.0.smithi059.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:37:24.225 INFO:tasks.workunit.client.0.smithi059.stdout: "os": "Linux", 2020-03-11T22:37:24.225 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_data": "td/divergent-priors/2", 2020-03-11T22:37:24.225 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:37:24.225 INFO:tasks.workunit.client.0.smithi059.stdout: "rotational": "1" 2020-03-11T22:37:24.225 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:37:24.225 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:37:24.226 INFO:tasks.workunit.client.0.smithi059.stdout: "osdmap_first_committed": 1, 2020-03-11T22:37:24.226 INFO:tasks.workunit.client.0.smithi059.stdout: "osdmap_last_committed": 40, 2020-03-11T22:37:24.226 INFO:tasks.workunit.client.0.smithi059.stdout: "crushmap": { 2020-03-11T22:37:24.226 INFO:tasks.workunit.client.0.smithi059.stdout: "devices": [ 2020-03-11T22:37:24.226 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.226 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 0, 2020-03-11T22:37:24.226 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "osd.0", 2020-03-11T22:37:24.227 INFO:tasks.workunit.client.0.smithi059.stdout: "class": "hdd" 2020-03-11T22:37:24.227 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.227 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.227 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 1, 2020-03-11T22:37:24.227 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "osd.1", 2020-03-11T22:37:24.227 INFO:tasks.workunit.client.0.smithi059.stdout: "class": "hdd" 2020-03-11T22:37:24.227 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.227 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.228 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 2, 2020-03-11T22:37:24.228 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "osd.2", 2020-03-11T22:37:24.228 INFO:tasks.workunit.client.0.smithi059.stdout: "class": "hdd" 2020-03-11T22:37:24.228 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:37:24.228 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:37:24.228 INFO:tasks.workunit.client.0.smithi059.stdout: "types": [ 2020-03-11T22:37:24.228 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.229 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 0, 2020-03-11T22:37:24.229 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "osd" 2020-03-11T22:37:24.229 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.229 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.229 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 1, 2020-03-11T22:37:24.229 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "host" 2020-03-11T22:37:24.229 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.229 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.230 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 2, 2020-03-11T22:37:24.230 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "chassis" 2020-03-11T22:37:24.230 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.230 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.230 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 3, 2020-03-11T22:37:24.230 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "rack" 2020-03-11T22:37:24.230 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.231 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.231 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 4, 2020-03-11T22:37:24.231 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "row" 2020-03-11T22:37:24.231 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.231 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.231 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 5, 2020-03-11T22:37:24.231 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "pdu" 2020-03-11T22:37:24.232 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.232 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.232 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 6, 2020-03-11T22:37:24.232 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "pod" 2020-03-11T22:37:24.232 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.232 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.233 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 7, 2020-03-11T22:37:24.233 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "room" 2020-03-11T22:37:24.233 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.233 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.233 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 8, 2020-03-11T22:37:24.233 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "datacenter" 2020-03-11T22:37:24.233 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.233 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.234 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 9, 2020-03-11T22:37:24.234 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "zone" 2020-03-11T22:37:24.234 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.234 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.234 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 10, 2020-03-11T22:37:24.234 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "region" 2020-03-11T22:37:24.234 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.235 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.235 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 11, 2020-03-11T22:37:24.235 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "root" 2020-03-11T22:37:24.235 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:37:24.235 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:37:24.235 INFO:tasks.workunit.client.0.smithi059.stdout: "buckets": [ 2020-03-11T22:37:24.235 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.235 INFO:tasks.workunit.client.0.smithi059.stdout: "id": -1, 2020-03-11T22:37:24.236 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "default", 2020-03-11T22:37:24.236 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 11, 2020-03-11T22:37:24.236 INFO:tasks.workunit.client.0.smithi059.stdout: "type_name": "root", 2020-03-11T22:37:24.236 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 19206, 2020-03-11T22:37:24.236 INFO:tasks.workunit.client.0.smithi059.stdout: "alg": "straw2", 2020-03-11T22:37:24.236 INFO:tasks.workunit.client.0.smithi059.stdout: "hash": "rjenkins1", 2020-03-11T22:37:24.236 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2020-03-11T22:37:24.237 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.237 INFO:tasks.workunit.client.0.smithi059.stdout: "id": -3, 2020-03-11T22:37:24.237 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 19206, 2020-03-11T22:37:24.237 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 0 2020-03-11T22:37:24.237 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:37:24.237 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:37:24.237 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.237 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.238 INFO:tasks.workunit.client.0.smithi059.stdout: "id": -2, 2020-03-11T22:37:24.238 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "default~hdd", 2020-03-11T22:37:24.238 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 11, 2020-03-11T22:37:24.238 INFO:tasks.workunit.client.0.smithi059.stdout: "type_name": "root", 2020-03-11T22:37:24.238 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 19206, 2020-03-11T22:37:24.238 INFO:tasks.workunit.client.0.smithi059.stdout: "alg": "straw2", 2020-03-11T22:37:24.238 INFO:tasks.workunit.client.0.smithi059.stdout: "hash": "rjenkins1", 2020-03-11T22:37:24.239 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2020-03-11T22:37:24.239 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.239 INFO:tasks.workunit.client.0.smithi059.stdout: "id": -4, 2020-03-11T22:37:24.239 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 19206, 2020-03-11T22:37:24.239 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 0 2020-03-11T22:37:24.239 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:37:24.239 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:37:24.239 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.240 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.240 INFO:tasks.workunit.client.0.smithi059.stdout: "id": -3, 2020-03-11T22:37:24.240 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "smithi059", 2020-03-11T22:37:24.240 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 1, 2020-03-11T22:37:24.240 INFO:tasks.workunit.client.0.smithi059.stdout: "type_name": "host", 2020-03-11T22:37:24.240 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 19206, 2020-03-11T22:37:24.240 INFO:tasks.workunit.client.0.smithi059.stdout: "alg": "straw2", 2020-03-11T22:37:24.241 INFO:tasks.workunit.client.0.smithi059.stdout: "hash": "rjenkins1", 2020-03-11T22:37:24.241 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2020-03-11T22:37:24.241 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.241 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 0, 2020-03-11T22:37:24.241 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 6402, 2020-03-11T22:37:24.241 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 0 2020-03-11T22:37:24.241 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.241 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.242 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 1, 2020-03-11T22:37:24.242 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 6402, 2020-03-11T22:37:24.242 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 1 2020-03-11T22:37:24.242 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.242 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.242 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 2, 2020-03-11T22:37:24.242 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 6402, 2020-03-11T22:37:24.243 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 2 2020-03-11T22:37:24.243 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:37:24.243 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:37:24.243 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.243 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.243 INFO:tasks.workunit.client.0.smithi059.stdout: "id": -4, 2020-03-11T22:37:24.243 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "smithi059~hdd", 2020-03-11T22:37:24.243 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 1, 2020-03-11T22:37:24.244 INFO:tasks.workunit.client.0.smithi059.stdout: "type_name": "host", 2020-03-11T22:37:24.244 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 19206, 2020-03-11T22:37:24.244 INFO:tasks.workunit.client.0.smithi059.stdout: "alg": "straw2", 2020-03-11T22:37:24.244 INFO:tasks.workunit.client.0.smithi059.stdout: "hash": "rjenkins1", 2020-03-11T22:37:24.244 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2020-03-11T22:37:24.244 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.244 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 0, 2020-03-11T22:37:24.245 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 6402, 2020-03-11T22:37:24.245 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 0 2020-03-11T22:37:24.245 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.245 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.245 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 1, 2020-03-11T22:37:24.245 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 6402, 2020-03-11T22:37:24.245 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 1 2020-03-11T22:37:24.246 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.246 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.246 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 2, 2020-03-11T22:37:24.246 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 6402, 2020-03-11T22:37:24.246 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 2 2020-03-11T22:37:24.246 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:37:24.247 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:37:24.247 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:37:24.247 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:37:24.247 INFO:tasks.workunit.client.0.smithi059.stdout: "rules": [ 2020-03-11T22:37:24.247 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.247 INFO:tasks.workunit.client.0.smithi059.stdout: "rule_id": 0, 2020-03-11T22:37:24.248 INFO:tasks.workunit.client.0.smithi059.stdout: "rule_name": "replicated_rule", 2020-03-11T22:37:24.248 INFO:tasks.workunit.client.0.smithi059.stdout: "ruleset": 0, 2020-03-11T22:37:24.248 INFO:tasks.workunit.client.0.smithi059.stdout: "type": 1, 2020-03-11T22:37:24.248 INFO:tasks.workunit.client.0.smithi059.stdout: "min_size": 1, 2020-03-11T22:37:24.248 INFO:tasks.workunit.client.0.smithi059.stdout: "max_size": 10, 2020-03-11T22:37:24.248 INFO:tasks.workunit.client.0.smithi059.stdout: "steps": [ 2020-03-11T22:37:24.248 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.248 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "take", 2020-03-11T22:37:24.249 INFO:tasks.workunit.client.0.smithi059.stdout: "item": -1, 2020-03-11T22:37:24.249 INFO:tasks.workunit.client.0.smithi059.stdout: "item_name": "default" 2020-03-11T22:37:24.249 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.249 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.249 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "choose_firstn", 2020-03-11T22:37:24.249 INFO:tasks.workunit.client.0.smithi059.stdout: "num": 0, 2020-03-11T22:37:24.249 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "osd" 2020-03-11T22:37:24.250 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.250 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.250 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "emit" 2020-03-11T22:37:24.250 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:37:24.250 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:37:24.250 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:37:24.250 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:37:24.251 INFO:tasks.workunit.client.0.smithi059.stdout: "tunables": { 2020-03-11T22:37:24.251 INFO:tasks.workunit.client.0.smithi059.stdout: "choose_local_tries": 0, 2020-03-11T22:37:24.251 INFO:tasks.workunit.client.0.smithi059.stdout: "choose_local_fallback_tries": 0, 2020-03-11T22:37:24.251 INFO:tasks.workunit.client.0.smithi059.stdout: "choose_total_tries": 50, 2020-03-11T22:37:24.251 INFO:tasks.workunit.client.0.smithi059.stdout: "chooseleaf_descend_once": 1, 2020-03-11T22:37:24.251 INFO:tasks.workunit.client.0.smithi059.stdout: "chooseleaf_vary_r": 1, 2020-03-11T22:37:24.251 INFO:tasks.workunit.client.0.smithi059.stdout: "chooseleaf_stable": 1, 2020-03-11T22:37:24.251 INFO:tasks.workunit.client.0.smithi059.stdout: "straw_calc_version": 1, 2020-03-11T22:37:24.252 INFO:tasks.workunit.client.0.smithi059.stdout: "allowed_bucket_algs": 54, 2020-03-11T22:37:24.252 INFO:tasks.workunit.client.0.smithi059.stdout: "profile": "jewel", 2020-03-11T22:37:24.252 INFO:tasks.workunit.client.0.smithi059.stdout: "optimal_tunables": 1, 2020-03-11T22:37:24.252 INFO:tasks.workunit.client.0.smithi059.stdout: "legacy_tunables": 0, 2020-03-11T22:37:24.252 INFO:tasks.workunit.client.0.smithi059.stdout: "minimum_required_version": "jewel", 2020-03-11T22:37:24.252 INFO:tasks.workunit.client.0.smithi059.stdout: "require_feature_tunables": 1, 2020-03-11T22:37:24.252 INFO:tasks.workunit.client.0.smithi059.stdout: "require_feature_tunables2": 1, 2020-03-11T22:37:24.253 INFO:tasks.workunit.client.0.smithi059.stdout: "has_v2_rules": 0, 2020-03-11T22:37:24.253 INFO:tasks.workunit.client.0.smithi059.stdout: "require_feature_tunables3": 1, 2020-03-11T22:37:24.253 INFO:tasks.workunit.client.0.smithi059.stdout: "has_v3_rules": 0, 2020-03-11T22:37:24.253 INFO:tasks.workunit.client.0.smithi059.stdout: "has_v4_buckets": 1, 2020-03-11T22:37:24.253 INFO:tasks.workunit.client.0.smithi059.stdout: "require_feature_tunables5": 1, 2020-03-11T22:37:24.253 INFO:tasks.workunit.client.0.smithi059.stdout: "has_v5_rules": 0 2020-03-11T22:37:24.253 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.254 INFO:tasks.workunit.client.0.smithi059.stdout: "choose_args": {} 2020-03-11T22:37:24.254 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.254 INFO:tasks.workunit.client.0.smithi059.stdout: "fsmap": { 2020-03-11T22:37:24.254 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": 1, 2020-03-11T22:37:24.254 INFO:tasks.workunit.client.0.smithi059.stdout: "default_fscid": -1, 2020-03-11T22:37:24.254 INFO:tasks.workunit.client.0.smithi059.stdout: "compat": { 2020-03-11T22:37:24.254 INFO:tasks.workunit.client.0.smithi059.stdout: "compat": {}, 2020-03-11T22:37:24.255 INFO:tasks.workunit.client.0.smithi059.stdout: "ro_compat": {}, 2020-03-11T22:37:24.255 INFO:tasks.workunit.client.0.smithi059.stdout: "incompat": { 2020-03-11T22:37:24.255 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_1": "base v0.20", 2020-03-11T22:37:24.255 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_2": "client writeable ranges", 2020-03-11T22:37:24.255 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_3": "default file layouts on dirs", 2020-03-11T22:37:24.255 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_4": "dir inode in separate object", 2020-03-11T22:37:24.255 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_5": "mds uses versioned encoding", 2020-03-11T22:37:24.256 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_6": "dirfrag is stored in omap", 2020-03-11T22:37:24.256 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_8": "no anchor table", 2020-03-11T22:37:24.256 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_9": "file layout v2", 2020-03-11T22:37:24.256 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_10": "snaprealm v2" 2020-03-11T22:37:24.256 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:37:24.256 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.256 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_flags": { 2020-03-11T22:37:24.256 INFO:tasks.workunit.client.0.smithi059.stdout: "enable_multiple": false, 2020-03-11T22:37:24.257 INFO:tasks.workunit.client.0.smithi059.stdout: "ever_enabled_multiple": false 2020-03-11T22:37:24.257 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.257 INFO:tasks.workunit.client.0.smithi059.stdout: "standbys": [], 2020-03-11T22:37:24.257 INFO:tasks.workunit.client.0.smithi059.stdout: "filesystems": [] 2020-03-11T22:37:24.257 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.257 INFO:tasks.workunit.client.0.smithi059.stdout: "mdsmap_first_committed": 1, 2020-03-11T22:37:24.257 INFO:tasks.workunit.client.0.smithi059.stdout: "mdsmap_last_committed": 1, 2020-03-11T22:37:24.258 INFO:tasks.workunit.client.0.smithi059.stdout: "auth": { 2020-03-11T22:37:24.258 INFO:tasks.workunit.client.0.smithi059.stdout: "first_committed": 1, 2020-03-11T22:37:24.258 INFO:tasks.workunit.client.0.smithi059.stdout: "last_committed": 5, 2020-03-11T22:37:24.258 INFO:tasks.workunit.client.0.smithi059.stdout: "num_secrets": 3 2020-03-11T22:37:24.258 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.258 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pg": 1, 2020-03-11T22:37:24.258 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pg_active": 1, 2020-03-11T22:37:24.258 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pg_unknown": 0, 2020-03-11T22:37:24.259 INFO:tasks.workunit.client.0.smithi059.stdout: "num_osd": 3, 2020-03-11T22:37:24.259 INFO:tasks.workunit.client.0.smithi059.stdout: "pool_sum": { 2020-03-11T22:37:24.259 INFO:tasks.workunit.client.0.smithi059.stdout: "stat_sum": { 2020-03-11T22:37:24.259 INFO:tasks.workunit.client.0.smithi059.stdout: "num_bytes": 47947, 2020-03-11T22:37:24.259 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects": 100, 2020-03-11T22:37:24.259 INFO:tasks.workunit.client.0.smithi059.stdout: "num_object_clones": 0, 2020-03-11T22:37:24.260 INFO:tasks.workunit.client.0.smithi059.stdout: "num_object_copies": 300, 2020-03-11T22:37:24.260 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:37:24.260 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_missing": 0, 2020-03-11T22:37:24.260 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_degraded": 100, 2020-03-11T22:37:24.260 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_misplaced": 0, 2020-03-11T22:37:24.260 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_unfound": 0, 2020-03-11T22:37:24.260 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_dirty": 100, 2020-03-11T22:37:24.261 INFO:tasks.workunit.client.0.smithi059.stdout: "num_whiteouts": 0, 2020-03-11T22:37:24.261 INFO:tasks.workunit.client.0.smithi059.stdout: "num_read": 0, 2020-03-11T22:37:24.261 INFO:tasks.workunit.client.0.smithi059.stdout: "num_read_kb": 0, 2020-03-11T22:37:24.261 INFO:tasks.workunit.client.0.smithi059.stdout: "num_write": 101, 2020-03-11T22:37:24.261 INFO:tasks.workunit.client.0.smithi059.stdout: "num_write_kb": 101, 2020-03-11T22:37:24.261 INFO:tasks.workunit.client.0.smithi059.stdout: "num_scrub_errors": 0, 2020-03-11T22:37:24.261 INFO:tasks.workunit.client.0.smithi059.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:37:24.262 INFO:tasks.workunit.client.0.smithi059.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:37:24.262 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_recovered": 0, 2020-03-11T22:37:24.262 INFO:tasks.workunit.client.0.smithi059.stdout: "num_bytes_recovered": 0, 2020-03-11T22:37:24.262 INFO:tasks.workunit.client.0.smithi059.stdout: "num_keys_recovered": 0, 2020-03-11T22:37:24.262 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_omap": 0, 2020-03-11T22:37:24.262 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:37:24.262 INFO:tasks.workunit.client.0.smithi059.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:37:24.263 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush": 0, 2020-03-11T22:37:24.263 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush_kb": 0, 2020-03-11T22:37:24.263 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict": 0, 2020-03-11T22:37:24.263 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict_kb": 0, 2020-03-11T22:37:24.263 INFO:tasks.workunit.client.0.smithi059.stdout: "num_promote": 0, 2020-03-11T22:37:24.263 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush_mode_high": 0, 2020-03-11T22:37:24.263 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush_mode_low": 0, 2020-03-11T22:37:24.264 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict_mode_some": 0, 2020-03-11T22:37:24.264 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict_mode_full": 0, 2020-03-11T22:37:24.264 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_pinned": 0, 2020-03-11T22:37:24.264 INFO:tasks.workunit.client.0.smithi059.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:37:24.264 INFO:tasks.workunit.client.0.smithi059.stdout: "num_large_omap_objects": 0, 2020-03-11T22:37:24.264 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_manifest": 0, 2020-03-11T22:37:24.264 INFO:tasks.workunit.client.0.smithi059.stdout: "num_omap_bytes": 0, 2020-03-11T22:37:24.264 INFO:tasks.workunit.client.0.smithi059.stdout: "num_omap_keys": 0, 2020-03-11T22:37:24.265 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_repaired": 0 2020-03-11T22:37:24.265 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.265 INFO:tasks.workunit.client.0.smithi059.stdout: "store_stats": { 2020-03-11T22:37:24.265 INFO:tasks.workunit.client.0.smithi059.stdout: "total": 0, 2020-03-11T22:37:24.265 INFO:tasks.workunit.client.0.smithi059.stdout: "available": 0, 2020-03-11T22:37:24.265 INFO:tasks.workunit.client.0.smithi059.stdout: "internally_reserved": 0, 2020-03-11T22:37:24.265 INFO:tasks.workunit.client.0.smithi059.stdout: "allocated": 0, 2020-03-11T22:37:24.266 INFO:tasks.workunit.client.0.smithi059.stdout: "data_stored": 0, 2020-03-11T22:37:24.266 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed": 0, 2020-03-11T22:37:24.266 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_allocated": 0, 2020-03-11T22:37:24.266 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_original": 0, 2020-03-11T22:37:24.266 INFO:tasks.workunit.client.0.smithi059.stdout: "omap_allocated": 0, 2020-03-11T22:37:24.266 INFO:tasks.workunit.client.0.smithi059.stdout: "internal_metadata": 0 2020-03-11T22:37:24.266 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.266 INFO:tasks.workunit.client.0.smithi059.stdout: "log_size": 17, 2020-03-11T22:37:24.267 INFO:tasks.workunit.client.0.smithi059.stdout: "ondisk_log_size": 17, 2020-03-11T22:37:24.267 INFO:tasks.workunit.client.0.smithi059.stdout: "up": 2, 2020-03-11T22:37:24.267 INFO:tasks.workunit.client.0.smithi059.stdout: "acting": 2, 2020-03-11T22:37:24.267 INFO:tasks.workunit.client.0.smithi059.stdout: "num_store_stats": 0 2020-03-11T22:37:24.267 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.267 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_sum": { 2020-03-11T22:37:24.267 INFO:tasks.workunit.client.0.smithi059.stdout: "up_from": 0, 2020-03-11T22:37:24.268 INFO:tasks.workunit.client.0.smithi059.stdout: "seq": 0, 2020-03-11T22:37:24.268 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pgs": 3, 2020-03-11T22:37:24.268 INFO:tasks.workunit.client.0.smithi059.stdout: "num_osds": 3, 2020-03-11T22:37:24.268 INFO:tasks.workunit.client.0.smithi059.stdout: "num_per_pool_osds": 3, 2020-03-11T22:37:24.268 INFO:tasks.workunit.client.0.smithi059.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:37:24.268 INFO:tasks.workunit.client.0.smithi059.stdout: "kb": 314572800, 2020-03-11T22:37:24.268 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used": 3179328, 2020-03-11T22:37:24.269 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used_data": 33280, 2020-03-11T22:37:24.269 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used_omap": 13, 2020-03-11T22:37:24.269 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used_meta": 3145714, 2020-03-11T22:37:24.269 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_avail": 311393472, 2020-03-11T22:37:24.269 INFO:tasks.workunit.client.0.smithi059.stdout: "statfs": { 2020-03-11T22:37:24.269 INFO:tasks.workunit.client.0.smithi059.stdout: "total": 322122547200, 2020-03-11T22:37:24.269 INFO:tasks.workunit.client.0.smithi059.stdout: "available": 318866915328, 2020-03-11T22:37:24.269 INFO:tasks.workunit.client.0.smithi059.stdout: "internally_reserved": 3221225472, 2020-03-11T22:37:24.270 INFO:tasks.workunit.client.0.smithi059.stdout: "allocated": 34078720, 2020-03-11T22:37:24.270 INFO:tasks.workunit.client.0.smithi059.stdout: "data_stored": 406500, 2020-03-11T22:37:24.270 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed": 0, 2020-03-11T22:37:24.270 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_allocated": 0, 2020-03-11T22:37:24.270 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_original": 0, 2020-03-11T22:37:24.270 INFO:tasks.workunit.client.0.smithi059.stdout: "omap_allocated": 14150, 2020-03-11T22:37:24.270 INFO:tasks.workunit.client.0.smithi059.stdout: "internal_metadata": 3221211322 2020-03-11T22:37:24.271 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.271 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_peers": [], 2020-03-11T22:37:24.271 INFO:tasks.workunit.client.0.smithi059.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:37:24.271 INFO:tasks.workunit.client.0.smithi059.stdout: "num_snap_trimming": 0, 2020-03-11T22:37:24.271 INFO:tasks.workunit.client.0.smithi059.stdout: "num_shards_repaired": 0, 2020-03-11T22:37:24.271 INFO:tasks.workunit.client.0.smithi059.stdout: "op_queue_age_hist": { 2020-03-11T22:37:24.271 INFO:tasks.workunit.client.0.smithi059.stdout: "histogram": [], 2020-03-11T22:37:24.271 INFO:tasks.workunit.client.0.smithi059.stdout: "upper_bound": 1 2020-03-11T22:37:24.272 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.272 INFO:tasks.workunit.client.0.smithi059.stdout: "perf_stat": { 2020-03-11T22:37:24.272 INFO:tasks.workunit.client.0.smithi059.stdout: "commit_latency_ms": 0, 2020-03-11T22:37:24.272 INFO:tasks.workunit.client.0.smithi059.stdout: "apply_latency_ms": 0, 2020-03-11T22:37:24.272 INFO:tasks.workunit.client.0.smithi059.stdout: "commit_latency_ns": 0, 2020-03-11T22:37:24.272 INFO:tasks.workunit.client.0.smithi059.stdout: "apply_latency_ns": 0 2020-03-11T22:37:24.272 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.273 INFO:tasks.workunit.client.0.smithi059.stdout: "alerts": [], 2020-03-11T22:37:24.273 INFO:tasks.workunit.client.0.smithi059.stdout: "network_ping_times": [] 2020-03-11T22:37:24.273 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.273 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_sum_by_class": { 2020-03-11T22:37:24.273 INFO:tasks.workunit.client.0.smithi059.stdout: "hdd": { 2020-03-11T22:37:24.273 INFO:tasks.workunit.client.0.smithi059.stdout: "up_from": 0, 2020-03-11T22:37:24.274 INFO:tasks.workunit.client.0.smithi059.stdout: "seq": 0, 2020-03-11T22:37:24.274 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pgs": 3, 2020-03-11T22:37:24.274 INFO:tasks.workunit.client.0.smithi059.stdout: "num_osds": 3, 2020-03-11T22:37:24.274 INFO:tasks.workunit.client.0.smithi059.stdout: "num_per_pool_osds": 3, 2020-03-11T22:37:24.274 INFO:tasks.workunit.client.0.smithi059.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:37:24.274 INFO:tasks.workunit.client.0.smithi059.stdout: "kb": 314572800, 2020-03-11T22:37:24.274 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used": 3179328, 2020-03-11T22:37:24.275 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used_data": 33280, 2020-03-11T22:37:24.275 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used_omap": 13, 2020-03-11T22:37:24.275 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used_meta": 3145714, 2020-03-11T22:37:24.275 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_avail": 311393472, 2020-03-11T22:37:24.275 INFO:tasks.workunit.client.0.smithi059.stdout: "statfs": { 2020-03-11T22:37:24.275 INFO:tasks.workunit.client.0.smithi059.stdout: "total": 322122547200, 2020-03-11T22:37:24.275 INFO:tasks.workunit.client.0.smithi059.stdout: "available": 318866915328, 2020-03-11T22:37:24.275 INFO:tasks.workunit.client.0.smithi059.stdout: "internally_reserved": 3221225472, 2020-03-11T22:37:24.276 INFO:tasks.workunit.client.0.smithi059.stdout: "allocated": 34078720, 2020-03-11T22:37:24.276 INFO:tasks.workunit.client.0.smithi059.stdout: "data_stored": 406500, 2020-03-11T22:37:24.276 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed": 0, 2020-03-11T22:37:24.276 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_allocated": 0, 2020-03-11T22:37:24.276 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_original": 0, 2020-03-11T22:37:24.276 INFO:tasks.workunit.client.0.smithi059.stdout: "omap_allocated": 14150, 2020-03-11T22:37:24.276 INFO:tasks.workunit.client.0.smithi059.stdout: "internal_metadata": 3221211322 2020-03-11T22:37:24.277 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.277 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_peers": [], 2020-03-11T22:37:24.277 INFO:tasks.workunit.client.0.smithi059.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:37:24.277 INFO:tasks.workunit.client.0.smithi059.stdout: "num_snap_trimming": 0, 2020-03-11T22:37:24.277 INFO:tasks.workunit.client.0.smithi059.stdout: "num_shards_repaired": 0, 2020-03-11T22:37:24.277 INFO:tasks.workunit.client.0.smithi059.stdout: "op_queue_age_hist": { 2020-03-11T22:37:24.277 INFO:tasks.workunit.client.0.smithi059.stdout: "histogram": [], 2020-03-11T22:37:24.278 INFO:tasks.workunit.client.0.smithi059.stdout: "upper_bound": 1 2020-03-11T22:37:24.278 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.278 INFO:tasks.workunit.client.0.smithi059.stdout: "perf_stat": { 2020-03-11T22:37:24.278 INFO:tasks.workunit.client.0.smithi059.stdout: "commit_latency_ms": 0, 2020-03-11T22:37:24.278 INFO:tasks.workunit.client.0.smithi059.stdout: "apply_latency_ms": 0, 2020-03-11T22:37:24.278 INFO:tasks.workunit.client.0.smithi059.stdout: "commit_latency_ns": 0, 2020-03-11T22:37:24.278 INFO:tasks.workunit.client.0.smithi059.stdout: "apply_latency_ns": 0 2020-03-11T22:37:24.278 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.279 INFO:tasks.workunit.client.0.smithi059.stdout: "alerts": [], 2020-03-11T22:37:24.279 INFO:tasks.workunit.client.0.smithi059.stdout: "network_ping_times": [] 2020-03-11T22:37:24.279 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:37:24.279 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.279 INFO:tasks.workunit.client.0.smithi059.stdout: "pool_stats": [ 2020-03-11T22:37:24.279 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.279 INFO:tasks.workunit.client.0.smithi059.stdout: "poolid": 1, 2020-03-11T22:37:24.280 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pg": 1, 2020-03-11T22:37:24.280 INFO:tasks.workunit.client.0.smithi059.stdout: "stat_sum": { 2020-03-11T22:37:24.280 INFO:tasks.workunit.client.0.smithi059.stdout: "num_bytes": 47947, 2020-03-11T22:37:24.280 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects": 100, 2020-03-11T22:37:24.280 INFO:tasks.workunit.client.0.smithi059.stdout: "num_object_clones": 0, 2020-03-11T22:37:24.280 INFO:tasks.workunit.client.0.smithi059.stdout: "num_object_copies": 300, 2020-03-11T22:37:24.280 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:37:24.280 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_missing": 0, 2020-03-11T22:37:24.281 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_degraded": 100, 2020-03-11T22:37:24.281 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_misplaced": 0, 2020-03-11T22:37:24.281 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_unfound": 0, 2020-03-11T22:37:24.281 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_dirty": 100, 2020-03-11T22:37:24.281 INFO:tasks.workunit.client.0.smithi059.stdout: "num_whiteouts": 0, 2020-03-11T22:37:24.281 INFO:tasks.workunit.client.0.smithi059.stdout: "num_read": 0, 2020-03-11T22:37:24.281 INFO:tasks.workunit.client.0.smithi059.stdout: "num_read_kb": 0, 2020-03-11T22:37:24.282 INFO:tasks.workunit.client.0.smithi059.stdout: "num_write": 101, 2020-03-11T22:37:24.282 INFO:tasks.workunit.client.0.smithi059.stdout: "num_write_kb": 101, 2020-03-11T22:37:24.282 INFO:tasks.workunit.client.0.smithi059.stdout: "num_scrub_errors": 0, 2020-03-11T22:37:24.282 INFO:tasks.workunit.client.0.smithi059.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:37:24.282 INFO:tasks.workunit.client.0.smithi059.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:37:24.282 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_recovered": 0, 2020-03-11T22:37:24.282 INFO:tasks.workunit.client.0.smithi059.stdout: "num_bytes_recovered": 0, 2020-03-11T22:37:24.282 INFO:tasks.workunit.client.0.smithi059.stdout: "num_keys_recovered": 0, 2020-03-11T22:37:24.283 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_omap": 0, 2020-03-11T22:37:24.283 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:37:24.283 INFO:tasks.workunit.client.0.smithi059.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:37:24.283 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush": 0, 2020-03-11T22:37:24.283 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush_kb": 0, 2020-03-11T22:37:24.283 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict": 0, 2020-03-11T22:37:24.283 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict_kb": 0, 2020-03-11T22:37:24.284 INFO:tasks.workunit.client.0.smithi059.stdout: "num_promote": 0, 2020-03-11T22:37:24.284 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush_mode_high": 0, 2020-03-11T22:37:24.284 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush_mode_low": 0, 2020-03-11T22:37:24.284 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict_mode_some": 0, 2020-03-11T22:37:24.284 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict_mode_full": 0, 2020-03-11T22:37:24.284 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_pinned": 0, 2020-03-11T22:37:24.284 INFO:tasks.workunit.client.0.smithi059.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:37:24.284 INFO:tasks.workunit.client.0.smithi059.stdout: "num_large_omap_objects": 0, 2020-03-11T22:37:24.285 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_manifest": 0, 2020-03-11T22:37:24.285 INFO:tasks.workunit.client.0.smithi059.stdout: "num_omap_bytes": 0, 2020-03-11T22:37:24.285 INFO:tasks.workunit.client.0.smithi059.stdout: "num_omap_keys": 0, 2020-03-11T22:37:24.285 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_repaired": 0 2020-03-11T22:37:24.285 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.285 INFO:tasks.workunit.client.0.smithi059.stdout: "store_stats": { 2020-03-11T22:37:24.285 INFO:tasks.workunit.client.0.smithi059.stdout: "total": 0, 2020-03-11T22:37:24.286 INFO:tasks.workunit.client.0.smithi059.stdout: "available": 0, 2020-03-11T22:37:24.286 INFO:tasks.workunit.client.0.smithi059.stdout: "internally_reserved": 0, 2020-03-11T22:37:24.286 INFO:tasks.workunit.client.0.smithi059.stdout: "allocated": 19529728, 2020-03-11T22:37:24.286 INFO:tasks.workunit.client.0.smithi059.stdout: "data_stored": 142522, 2020-03-11T22:37:24.286 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed": 0, 2020-03-11T22:37:24.286 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_allocated": 0, 2020-03-11T22:37:24.286 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_original": 0, 2020-03-11T22:37:24.286 INFO:tasks.workunit.client.0.smithi059.stdout: "omap_allocated": 0, 2020-03-11T22:37:24.287 INFO:tasks.workunit.client.0.smithi059.stdout: "internal_metadata": 0 2020-03-11T22:37:24.287 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.287 INFO:tasks.workunit.client.0.smithi059.stdout: "log_size": 17, 2020-03-11T22:37:24.287 INFO:tasks.workunit.client.0.smithi059.stdout: "ondisk_log_size": 17, 2020-03-11T22:37:24.287 INFO:tasks.workunit.client.0.smithi059.stdout: "up": 2, 2020-03-11T22:37:24.287 INFO:tasks.workunit.client.0.smithi059.stdout: "acting": 2, 2020-03-11T22:37:24.287 INFO:tasks.workunit.client.0.smithi059.stdout: "num_store_stats": 3 2020-03-11T22:37:24.288 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:37:24.288 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:37:24.288 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_stats": [ 2020-03-11T22:37:24.288 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.288 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 0, 2020-03-11T22:37:24.288 INFO:tasks.workunit.client.0.smithi059.stdout: "seq": 146028888084 2020-03-11T22:37:24.288 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.288 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.289 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 1, 2020-03-11T22:37:24.289 INFO:tasks.workunit.client.0.smithi059.stdout: "seq": 42949672975 2020-03-11T22:37:24.289 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.289 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.289 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 2, 2020-03-11T22:37:24.289 INFO:tasks.workunit.client.0.smithi059.stdout: "seq": 150323855379 2020-03-11T22:37:24.289 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:37:24.290 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:37:24.290 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pg_by_state": [ 2020-03-11T22:37:24.290 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.290 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "active+undersized+degraded", 2020-03-11T22:37:24.290 INFO:tasks.workunit.client.0.smithi059.stdout: "num": 1 2020-03-11T22:37:24.290 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:37:24.290 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:37:24.290 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pg_by_osd": [ 2020-03-11T22:37:24.291 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.291 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 2, 2020-03-11T22:37:24.291 INFO:tasks.workunit.client.0.smithi059.stdout: "num_primary_pg": 0, 2020-03-11T22:37:24.291 INFO:tasks.workunit.client.0.smithi059.stdout: "num_acting_pg": 1, 2020-03-11T22:37:24.291 INFO:tasks.workunit.client.0.smithi059.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:37:24.291 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.291 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.292 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 1, 2020-03-11T22:37:24.292 INFO:tasks.workunit.client.0.smithi059.stdout: "num_primary_pg": 0, 2020-03-11T22:37:24.292 INFO:tasks.workunit.client.0.smithi059.stdout: "num_acting_pg": 0, 2020-03-11T22:37:24.292 INFO:tasks.workunit.client.0.smithi059.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:37:24.292 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.292 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.292 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 0, 2020-03-11T22:37:24.293 INFO:tasks.workunit.client.0.smithi059.stdout: "num_primary_pg": 1, 2020-03-11T22:37:24.293 INFO:tasks.workunit.client.0.smithi059.stdout: "num_acting_pg": 1, 2020-03-11T22:37:24.293 INFO:tasks.workunit.client.0.smithi059.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:37:24.293 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:37:24.293 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:37:24.293 INFO:tasks.workunit.client.0.smithi059.stdout: "purged_snaps": [ 2020-03-11T22:37:24.293 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:37:24.293 INFO:tasks.workunit.client.0.smithi059.stdout: "pool": 1, 2020-03-11T22:37:24.294 INFO:tasks.workunit.client.0.smithi059.stdout: "purged_snaps": {} 2020-03-11T22:37:24.294 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:37:24.294 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:37:24.294 INFO:tasks.workunit.client.0.smithi059.stdout: "servicemap": { 2020-03-11T22:37:24.294 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": 1, 2020-03-11T22:37:24.294 INFO:tasks.workunit.client.0.smithi059.stdout: "modified": "2020-03-11T22:35:07.878044+0000", 2020-03-11T22:37:24.294 INFO:tasks.workunit.client.0.smithi059.stdout: "services": {} 2020-03-11T22:37:24.295 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:37:24.295 INFO:tasks.workunit.client.0.smithi059.stdout: "paxos": { 2020-03-11T22:37:24.295 INFO:tasks.workunit.client.0.smithi059.stdout: "first_committed": 1, 2020-03-11T22:37:24.295 INFO:tasks.workunit.client.0.smithi059.stdout: "last_committed": 390, 2020-03-11T22:37:24.295 INFO:tasks.workunit.client.0.smithi059.stdout: "last_pn": 0, 2020-03-11T22:37:24.295 INFO:tasks.workunit.client.0.smithi059.stdout: "accepted_pn": 0 2020-03-11T22:37:24.295 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:37:24.295 INFO:tasks.workunit.client.0.smithi059.stdout:} 2020-03-11T22:37:24.296 INFO:tasks.workunit.client.0.smithi059.stderr:report 437111840 2020-03-11T22:37:24.296 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: return 1 2020-03-11T22:37:24.297 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:536: TEST_divergent_2: echo 'delay recovery' 2020-03-11T22:37:24.297 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:537: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:37:24.297 INFO:tasks.workunit.client.0.smithi059.stdout:delay recovery 2020-03-11T22:37:24.490 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:37:24.491 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2020-03-11T22:36:27.936281+0000 40'101 40:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:35:29.715580+0000 0'0 2020-03-11T22:35:29.715580+0000 0 2020-03-11T22:37:24.491 INFO:tasks.workunit.client.0.smithi059.stdout: 2020-03-11T22:37:24.491 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:37:24.492 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2020-03-11T22:37:24.500 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:538: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:37:24.501 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: get_asok_path osd.0 2020-03-11T22:37:24.501 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-11T22:37:24.502 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-11T22:37:24.502 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:37:24.502 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:37:24.502 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:37:24.502 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17931/ceph-osd.0.asok 2020-03-11T22:37:24.502 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: CEPH_ARGS= 2020-03-11T22:37:24.502 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.17931/ceph-osd.0.asok set_recovery_delay 100000 2020-03-11T22:37:24.598 INFO:tasks.workunit.client.0.smithi059.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:37:24.606 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:538: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:37:24.606 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: get_asok_path osd.2 2020-03-11T22:37:24.606 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-03-11T22:37:24.607 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-03-11T22:37:24.607 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:37:24.607 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:37:24.607 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:37:24.607 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17931/ceph-osd.2.asok 2020-03-11T22:37:24.608 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: CEPH_ARGS= 2020-03-11T22:37:24.608 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.17931/ceph-osd.2.asok set_recovery_delay 100000 2020-03-11T22:37:24.703 INFO:tasks.workunit.client.0.smithi059.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:37:24.710 INFO:tasks.workunit.client.0.smithi059.stdout:revive divergent 1 2020-03-11T22:37:24.711 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:544: TEST_divergent_2: echo 'revive divergent 1' 2020-03-11T22:37:24.711 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:545: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:37:25.053 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:37:25.054 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2020-03-11T22:36:27.936281+0000 40'101 40:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:35:29.715580+0000 0'0 2020-03-11T22:35:29.715580+0000 0 2020-03-11T22:37:25.054 INFO:tasks.workunit.client.0.smithi059.stdout: 2020-03-11T22:37:25.054 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:37:25.055 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2020-03-11T22:37:25.063 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:546: TEST_divergent_2: ceph osd set noup 2020-03-11T22:37:25.740 INFO:tasks.workunit.client.0.smithi059.stderr:noup is set 2020-03-11T22:37:25.751 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:547: TEST_divergent_2: activate_osd td/divergent-priors 1 2020-03-11T22:37:25.752 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:37:25.752 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:37:25.752 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-11T22:37:25.752 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:37:25.752 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:37:25.752 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=cb7f2e42-1934-4145-a3c3-29bdd93f8adb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:37:25.753 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:37:25.753 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:37:25.753 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:37:25.753 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:37:25.753 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:37:25.753 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:37:25.754 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:37:25.754 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:37:25.754 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:37:25.754 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:37:25.754 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:37:25.754 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:37:25.754 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:37:25.755 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:37:25.755 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17931/$cluster-$name.asok' 2020-03-11T22:37:25.755 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' 2020-03-11T22:37:25.755 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:37:25.755 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:37:25.755 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:37:25.755 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:37:25.756 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:37:25.756 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:37:25.756 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:37:25.756 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:37:25.756 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:37:25.756 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-11T22:37:25.756 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=cb7f2e42-1934-4145-a3c3-29bdd93f8adb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:37:25.757 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-03-11T22:37:25.757 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.1 2020-03-11T22:37:25.757 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-11T22:37:25.758 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:37:25.758 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:37:25.758 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:37:25.780 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:37:25.776+0000 7efd5e54dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:25.782 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:37:25.778+0000 7efd5e54dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:25.784 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:37:25.780+0000 7efd5e54dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:26.125 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: return 0 2020-03-11T22:37:26.125 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:548: TEST_divergent_2: sleep 5 2020-03-11T22:37:26.299 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:37:26.296+0000 7efd5e54dec0 -1 Falling back to public interface 2020-03-11T22:37:27.166 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:37:27.162+0000 7efd5e54dec0 -1 osd.1 28 log_to_monitors {default=true} 2020-03-11T22:37:31.126 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:550: TEST_divergent_2: echo 'delay recovery divergent' 2020-03-11T22:37:31.127 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:551: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:37:31.127 INFO:tasks.workunit.client.0.smithi059.stdout:delay recovery divergent 2020-03-11T22:37:31.481 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:37:31.481 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2020-03-11T22:36:27.936281+0000 40'101 41:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:35:29.715580+0000 0'0 2020-03-11T22:35:29.715580+0000 0 2020-03-11T22:37:31.481 INFO:tasks.workunit.client.0.smithi059.stdout: 2020-03-11T22:37:31.481 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:37:31.482 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2020-03-11T22:37:31.491 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: get_asok_path osd.1 2020-03-11T22:37:31.492 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:37:31.492 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:37:31.492 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:37:31.492 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:37:31.492 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:37:31.492 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17931/ceph-osd.1.asok 2020-03-11T22:37:31.493 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: CEPH_ARGS= 2020-03-11T22:37:31.493 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.17931/ceph-osd.1.asok set_recovery_delay 100000 2020-03-11T22:37:31.588 INFO:tasks.workunit.client.0.smithi059.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:37:31.595 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:554: TEST_divergent_2: ceph osd unset noup 2020-03-11T22:37:32.378 INFO:tasks.workunit.client.0.smithi059.stderr:noup is unset 2020-03-11T22:37:32.387 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:556: TEST_divergent_2: wait_for_osd up 0 2020-03-11T22:37:32.387 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:37:32.387 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:37:32.387 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:37:32.388 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:37:32.388 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:37:32.388 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:37:32.388 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:37:32.388 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:37:32.388 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:37:32.749 INFO:tasks.workunit.client.0.smithi059.stdout:osd.0 up in weight 1 up_from 34 up_thru 39 down_at 29 last_clean_interval [5,28) [v2:127.0.0.1:6802/30799,v1:127.0.0.1:6803/30799] [v2:127.0.0.1:6804/30799,v1:127.0.0.1:6805/30799] exists,up 6b6c1a2d-a55b-410f-9b5c-d5f724d8ff3d 2020-03-11T22:37:32.749 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:37:32.749 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:37:32.750 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:37:32.750 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:37:32.750 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:557: TEST_divergent_2: wait_for_osd up 1 2020-03-11T22:37:32.750 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:37:32.750 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:37:32.750 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:37:32.750 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:37:32.750 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:37:32.751 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:37:32.751 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:37:32.751 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:37:33.180 INFO:tasks.workunit.client.0.smithi059.stdout:osd.1 up in weight 1 up_from 44 up_thru 44 down_at 30 last_clean_interval [10,28) [v2:127.0.0.1:6818/33540,v1:127.0.0.1:6819/33540] [v2:127.0.0.1:6820/33540,v1:127.0.0.1:6821/33540] exists,up 55fb3134-fe56-4d99-9135-8d0c47f417da 2020-03-11T22:37:33.180 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:37:33.181 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:37:33.181 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:37:33.181 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:37:33.181 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:558: TEST_divergent_2: wait_for_osd up 2 2020-03-11T22:37:33.181 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:37:33.181 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:37:33.182 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:37:33.182 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:37:33.182 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:37:33.182 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:37:33.182 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:37:33.182 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:37:33.566 INFO:tasks.workunit.client.0.smithi059.stdout:osd.2 up in weight 1 up_from 35 up_thru 0 down_at 31 last_clean_interval [15,28) [v2:127.0.0.1:6810/30993,v1:127.0.0.1:6811/30993] [v2:127.0.0.1:6812/30993,v1:127.0.0.1:6813/30993] exists,up 3aa1b594-c731-4355-b2e6-05a8c3d7bb4b 2020-03-11T22:37:33.566 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:37:33.567 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:37:33.567 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:37:33.567 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:560: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:37:33.953 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:37:33.953 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 100 0 0 0 0 46628 0 0 16 16 peering 2020-03-11T22:37:32.158639+0000 28'106 44:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:35:29.715580+0000 0'0 2020-03-11T22:35:29.715580+0000 0 2020-03-11T22:37:33.953 INFO:tasks.workunit.client.0.smithi059.stdout: 2020-03-11T22:37:33.953 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:37:33.955 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2020-03-11T22:37:33.963 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:561: TEST_divergent_2: echo 'wait for peering' 2020-03-11T22:37:33.963 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:562: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:37:33.963 INFO:tasks.workunit.client.0.smithi059.stdout:wait for peering 2020-03-11T22:37:34.304 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:37:34.304 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 100 0 0 0 0 46628 0 0 16 16 peering 2020-03-11T22:37:32.158639+0000 28'106 44:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:35:29.715580+0000 0'0 2020-03-11T22:35:29.715580+0000 0 2020-03-11T22:37:34.304 INFO:tasks.workunit.client.0.smithi059.stdout: 2020-03-11T22:37:34.304 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:37:34.306 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2020-03-11T22:37:34.313 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:563: TEST_divergent_2: rados -p test put foo /etc/fstab 2020-03-11T22:37:40.379 INFO:tasks.workunit.client.0.smithi059.stdout:killing divergent 1 2020-03-11T22:37:40.379 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:567: TEST_divergent_2: echo 'killing divergent 1' 2020-03-11T22:37:40.379 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:568: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:37:40.731 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:37:40.732 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 100 4 4 0 0 47947 0 0 11 11 active+recovering+degraded+wait 2020-03-11T22:37:32.500917+0000 40'101 45:125 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:35:29.715580+0000 0'0 2020-03-11T22:35:29.715580+0000 0 2020-03-11T22:37:40.732 INFO:tasks.workunit.client.0.smithi059.stdout: 2020-03-11T22:37:40.732 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:37:40.733 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2020-03-11T22:37:40.741 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:569: TEST_divergent_2: kill_daemons td/divergent-priors KILL osd.1 2020-03-11T22:37:40.742 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:37:40.742 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:37:40.742 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:37:40.742 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:37:40.742 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:37:40.855 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:37:40.855 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:572: TEST_divergent_2: expfile=td/divergent-priors/exp.17931.out 2020-03-11T22:37:40.855 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:573: TEST_divergent_2: _objectstore_tool_nodown td/divergent-priors 1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.17931.out 2020-03-11T22:37:40.855 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/divergent-priors 2020-03-11T22:37:40.855 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-03-11T22:37:40.855 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=1 2020-03-11T22:37:40.856 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-03-11T22:37:40.856 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2020-03-11T22:37:40.856 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.17931.out 2020-03-11T22:37:41.472 INFO:tasks.workunit.client.0.smithi059.stderr:Exporting 1.0 info 1.0( v 45'102 lc 0'0 (28'90,45'102] local-lis/les=44/45 n=101 ec=22/22 lis/c=44/22 les/c/f=45/23/0 sis=44 pruub=7.848089054s) 2020-03-11T22:37:41.484 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:023dd3d2:::existing_12:head# 2020-03-11T22:37:41.505 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:04c40a0c:::existing_93:head# 2020-03-11T22:37:41.518 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:064ff5fb:::existing_36:head# 2020-03-11T22:37:41.520 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:091f9f1d:::existing_94:head# 2020-03-11T22:37:41.521 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:0a865a93:::existing_54:head# 2020-03-11T22:37:41.526 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:0fed513c:::existing_41:head# 2020-03-11T22:37:41.527 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:13b9e0b8:::existing_92:head# 2020-03-11T22:37:41.534 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:169157fa:::existing_81:head# 2020-03-11T22:37:41.539 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:17731382:::existing_58:head# 2020-03-11T22:37:41.540 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:18a45a39:::existing_43:head# 2020-03-11T22:37:41.540 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:1c160f01:::existing_66:head# 2020-03-11T22:37:41.541 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:1dd6c249:::existing_97:head# 2020-03-11T22:37:41.541 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:227e0c77:::existing_7:head# 2020-03-11T22:37:41.541 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:23e1b063:::existing_59:head# 2020-03-11T22:37:41.542 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:28566edb:::existing_78:head# 2020-03-11T22:37:41.546 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:2a101182:::existing_20:head# 2020-03-11T22:37:41.548 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:2d1519da:::existing_96:head# 2020-03-11T22:37:41.549 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:3078b87a:::existing_86:head# 2020-03-11T22:37:41.550 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:30bde01b:::existing_15:head# 2020-03-11T22:37:41.550 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:317a7e19:::existing_56:head# 2020-03-11T22:37:41.551 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:38b9c7fa:::existing_33:head# 2020-03-11T22:37:41.553 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:3e16d84b:::existing_48:head# 2020-03-11T22:37:41.558 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:45fe8203:::existing_75:head# 2020-03-11T22:37:41.562 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:4e82c9ba:::existing_50:head# 2020-03-11T22:37:41.562 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:5298dcdb:::existing_57:head# 2020-03-11T22:37:41.563 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:52ebaa44:::existing_32:head# 2020-03-11T22:37:41.563 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:53588912:::existing_9:head# 2020-03-11T22:37:41.564 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:5b7e2de6:::existing_84:head# 2020-03-11T22:37:41.565 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:5da0255c:::existing_13:head# 2020-03-11T22:37:41.565 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:5da2dab8:::existing_53:head# 2020-03-11T22:37:41.566 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:5ee3514b:::existing_30:head# 2020-03-11T22:37:41.566 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:602f83fe:::foo:head# 2020-03-11T22:37:41.574 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:60426cc7:::existing_62:head# 2020-03-11T22:37:41.575 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:628918ca:::existing_79:head# 2020-03-11T22:37:41.575 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:63f8ab2d:::existing_77:head# 2020-03-11T22:37:41.576 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:67e1780f:::existing_27:head# 2020-03-11T22:37:41.576 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:68143ca1:::existing_71:head# 2020-03-11T22:37:41.587 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:6bac9509:::existing_29:head# 2020-03-11T22:37:41.587 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:6bbec798:::existing_45:head# 2020-03-11T22:37:41.587 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:6cb23a40:::existing_65:head# 2020-03-11T22:37:41.587 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:7055ec5e:::existing_83:head# 2020-03-11T22:37:41.588 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:70a54e2b:::existing_87:head# 2020-03-11T22:37:41.588 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:750ace21:::existing_19:head# 2020-03-11T22:37:41.590 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:78623b1d:::existing_6:head# 2020-03-11T22:37:41.590 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:79881152:::existing_63:head# 2020-03-11T22:37:41.591 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:7a80c0d5:::existing_99:head# 2020-03-11T22:37:41.592 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:81490677:::existing_76:head# 2020-03-11T22:37:41.592 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:85ec8e5d:::existing_85:head# 2020-03-11T22:37:41.593 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:8e2abbf0:::existing_49:head# 2020-03-11T22:37:41.593 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:8ee0fe0f:::existing_52:head# 2020-03-11T22:37:41.594 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:923c5d4d:::existing_5:head# 2020-03-11T22:37:41.594 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:92731e71:::existing_89:head# 2020-03-11T22:37:41.595 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:928205cf:::existing_82:head# 2020-03-11T22:37:41.595 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:93be2267:::existing_24:head# 2020-03-11T22:37:41.596 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:99ceda2c:::existing_44:head# 2020-03-11T22:37:41.596 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:9dd917ee:::existing_26:head# 2020-03-11T22:37:41.597 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:a28805a6:::existing_72:head# 2020-03-11T22:37:41.598 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:a294a8f6:::existing_16:head# 2020-03-11T22:37:41.598 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:a2f5473a:::existing_42:head# 2020-03-11T22:37:41.598 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:a504a533:::existing_90:head# 2020-03-11T22:37:41.599 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:a691337e:::existing_61:head# 2020-03-11T22:37:41.600 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:a6e8d7ac:::existing_28:head# 2020-03-11T22:37:41.600 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:a9fde86e:::existing_18:head# 2020-03-11T22:37:41.600 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:aa309faa:::existing_68:head# 2020-03-11T22:37:41.601 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:ae40b0ed:::existing_31:head# 2020-03-11T22:37:41.601 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:b279b1fe:::existing_10:head# 2020-03-11T22:37:41.602 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:b343e277:::existing_88:head# 2020-03-11T22:37:41.602 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:b7130e90:::existing_80:head# 2020-03-11T22:37:41.603 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:bc6f7309:::existing_70:head# 2020-03-11T22:37:41.603 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:be8489b7:::existing_38:head# 2020-03-11T22:37:41.604 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:beb4e5e3:::existing_47:head# 2020-03-11T22:37:41.605 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:bebec877:::existing_74:head# 2020-03-11T22:37:41.605 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:c04347d8:::existing_60:head# 2020-03-11T22:37:41.606 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:c156033e:::existing_100:head# 2020-03-11T22:37:41.607 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:c19f5a8e:::existing_8:head# 2020-03-11T22:37:41.607 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:c3e6ebe4:::existing_17:head# 2020-03-11T22:37:41.608 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:c6c5a7c2:::existing_39:head# 2020-03-11T22:37:41.608 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:cc1d0abf:::existing_95:head# 2020-03-11T22:37:41.609 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:ccb3865b:::existing_73:head# 2020-03-11T22:37:41.610 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:cd84e19e:::existing_40:head# 2020-03-11T22:37:41.610 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:d2cd64ce:::existing_37:head# 2020-03-11T22:37:41.612 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:d41a25c9:::existing_22:head# 2020-03-11T22:37:41.612 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:d7f088da:::existing_91:head# 2020-03-11T22:37:41.612 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:d9cbd1d6:::existing_64:head# 2020-03-11T22:37:41.612 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:dcc4ff82:::existing_98:head# 2020-03-11T22:37:41.613 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:dd677bd3:::existing_35:head# 2020-03-11T22:37:41.613 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:dd6d90e7:::existing_46:head# 2020-03-11T22:37:41.616 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:e0e22077:::existing_21:head# 2020-03-11T22:37:41.616 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:e255b710:::existing_55:head# 2020-03-11T22:37:41.616 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:eff1d5db:::existing_14:head# 2020-03-11T22:37:41.617 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:f1b156bf:::existing_23:head# 2020-03-11T22:37:41.617 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:f5ce4199:::existing_34:head# 2020-03-11T22:37:41.618 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:f6809456:::existing_11:head# 2020-03-11T22:37:41.619 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:f699a2a9:::existing_67:head# 2020-03-11T22:37:41.621 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:f9baac43:::existing_25:head# 2020-03-11T22:37:41.621 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:fb76688e:::existing_51:head# 2020-03-11T22:37:41.621 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:fbb92eec:::existing_69:head# 2020-03-11T22:37:41.621 INFO:tasks.workunit.client.0.smithi059.stderr:Export successful 2020-03-11T22:37:41.622 INFO:tasks.workunit.client.0.smithi059.stdout: marking collection for removal 2020-03-11T22:37:41.622 INFO:tasks.workunit.client.0.smithi059.stdout:setting '_remove' omap key 2020-03-11T22:37:41.622 INFO:tasks.workunit.client.0.smithi059.stdout:finish_remove_pgs 1.0_head removing 1.0 2020-03-11T22:37:41.702 INFO:tasks.workunit.client.0.smithi059.stderr:Remove successful 2020-03-11T22:37:42.137 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:574: TEST_divergent_2: _objectstore_tool_nodown td/divergent-priors 1 --op import --file td/divergent-priors/exp.17931.out 2020-03-11T22:37:42.137 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/divergent-priors 2020-03-11T22:37:42.138 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-03-11T22:37:42.138 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=1 2020-03-11T22:37:42.138 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-03-11T22:37:42.138 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2020-03-11T22:37:42.138 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op import --file td/divergent-priors/exp.17931.out 2020-03-11T22:37:42.634 INFO:tasks.workunit.client.0.smithi059.stdout:get_pg_num_history pg_num_history pg_num_history(e45 pg_nums {1={22=1}} deleted_pools ) 2020-03-11T22:37:42.634 INFO:tasks.workunit.client.0.smithi059.stdout:Importing pgid 1.0 2020-03-11T22:37:42.634 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:023dd3d2:::existing_12:head# 2020-03-11T22:37:42.634 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:42.676 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:04c40a0c:::existing_93:head# 2020-03-11T22:37:42.676 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:42.684 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:064ff5fb:::existing_36:head# 2020-03-11T22:37:42.684 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:42.692 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:091f9f1d:::existing_94:head# 2020-03-11T22:37:42.692 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:42.701 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:0a865a93:::existing_54:head# 2020-03-11T22:37:42.701 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:42.709 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:0fed513c:::existing_41:head# 2020-03-11T22:37:42.709 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:42.717 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:13b9e0b8:::existing_92:head# 2020-03-11T22:37:42.718 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:42.726 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:169157fa:::existing_81:head# 2020-03-11T22:37:42.726 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:42.734 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:17731382:::existing_58:head# 2020-03-11T22:37:42.734 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:42.742 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:18a45a39:::existing_43:head# 2020-03-11T22:37:42.742 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:42.751 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:1c160f01:::existing_66:head# 2020-03-11T22:37:42.751 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:42.759 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:1dd6c249:::existing_97:head# 2020-03-11T22:37:42.759 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:42.767 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:227e0c77:::existing_7:head# 2020-03-11T22:37:42.768 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:42.776 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:23e1b063:::existing_59:head# 2020-03-11T22:37:42.776 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:42.831 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:28566edb:::existing_78:head# 2020-03-11T22:37:42.831 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:42.840 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:2a101182:::existing_20:head# 2020-03-11T22:37:42.840 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:42.848 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:2d1519da:::existing_96:head# 2020-03-11T22:37:42.848 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:42.857 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:3078b87a:::existing_86:head# 2020-03-11T22:37:42.857 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:42.865 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:30bde01b:::existing_15:head# 2020-03-11T22:37:42.865 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:42.873 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:317a7e19:::existing_56:head# 2020-03-11T22:37:42.874 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:42.882 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:38b9c7fa:::existing_33:head# 2020-03-11T22:37:42.882 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:42.890 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:3e16d84b:::existing_48:head# 2020-03-11T22:37:42.890 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:42.898 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:45fe8203:::existing_75:head# 2020-03-11T22:37:42.899 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:42.907 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:4e82c9ba:::existing_50:head# 2020-03-11T22:37:42.907 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:42.915 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:5298dcdb:::existing_57:head# 2020-03-11T22:37:42.915 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:42.923 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:52ebaa44:::existing_32:head# 2020-03-11T22:37:42.924 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:42.932 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:53588912:::existing_9:head# 2020-03-11T22:37:42.932 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:42.940 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:5b7e2de6:::existing_84:head# 2020-03-11T22:37:42.940 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:42.964 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:5da0255c:::existing_13:head# 2020-03-11T22:37:42.965 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:42.976 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:5da2dab8:::existing_53:head# 2020-03-11T22:37:42.976 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:42.984 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:5ee3514b:::existing_30:head# 2020-03-11T22:37:42.984 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:42.992 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:602f83fe:::foo:head# 2020-03-11T22:37:42.993 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:43.001 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:60426cc7:::existing_62:head# 2020-03-11T22:37:43.001 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:43.009 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:628918ca:::existing_79:head# 2020-03-11T22:37:43.010 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:43.018 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:63f8ab2d:::existing_77:head# 2020-03-11T22:37:43.018 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:43.026 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:67e1780f:::existing_27:head# 2020-03-11T22:37:43.026 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:43.034 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:68143ca1:::existing_71:head# 2020-03-11T22:37:43.034 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:43.043 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:6bac9509:::existing_29:head# 2020-03-11T22:37:43.043 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:43.051 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:6bbec798:::existing_45:head# 2020-03-11T22:37:43.051 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:43.059 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:6cb23a40:::existing_65:head# 2020-03-11T22:37:43.059 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:43.091 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:7055ec5e:::existing_83:head# 2020-03-11T22:37:43.091 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:43.123 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:70a54e2b:::existing_87:head# 2020-03-11T22:37:43.123 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:43.134 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:750ace21:::existing_19:head# 2020-03-11T22:37:43.135 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:43.143 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:78623b1d:::existing_6:head# 2020-03-11T22:37:43.143 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:43.151 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:79881152:::existing_63:head# 2020-03-11T22:37:43.151 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:43.159 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:7a80c0d5:::existing_99:head# 2020-03-11T22:37:43.160 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:43.168 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:81490677:::existing_76:head# 2020-03-11T22:37:43.168 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:43.177 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:85ec8e5d:::existing_85:head# 2020-03-11T22:37:43.177 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:43.185 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:8e2abbf0:::existing_49:head# 2020-03-11T22:37:43.185 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:43.193 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:8ee0fe0f:::existing_52:head# 2020-03-11T22:37:43.193 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:43.201 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:923c5d4d:::existing_5:head# 2020-03-11T22:37:43.202 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:43.210 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:92731e71:::existing_89:head# 2020-03-11T22:37:43.210 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:43.218 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:928205cf:::existing_82:head# 2020-03-11T22:37:43.218 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:43.226 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:93be2267:::existing_24:head# 2020-03-11T22:37:43.227 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:43.235 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:99ceda2c:::existing_44:head# 2020-03-11T22:37:43.235 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:43.265 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:9dd917ee:::existing_26:head# 2020-03-11T22:37:43.265 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:43.276 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:a28805a6:::existing_72:head# 2020-03-11T22:37:43.277 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:43.285 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:a294a8f6:::existing_16:head# 2020-03-11T22:37:43.285 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:43.293 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:a2f5473a:::existing_42:head# 2020-03-11T22:37:43.293 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:43.302 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:a504a533:::existing_90:head# 2020-03-11T22:37:43.302 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:43.310 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:a691337e:::existing_61:head# 2020-03-11T22:37:43.310 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:43.318 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:a6e8d7ac:::existing_28:head# 2020-03-11T22:37:43.318 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:43.327 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:a9fde86e:::existing_18:head# 2020-03-11T22:37:43.327 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:43.335 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:aa309faa:::existing_68:head# 2020-03-11T22:37:43.335 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:43.343 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:ae40b0ed:::existing_31:head# 2020-03-11T22:37:43.343 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:43.419 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:b279b1fe:::existing_10:head# 2020-03-11T22:37:43.419 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:43.448 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:b343e277:::existing_88:head# 2020-03-11T22:37:43.448 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:43.452 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:b7130e90:::existing_80:head# 2020-03-11T22:37:43.452 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:43.481 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:bc6f7309:::existing_70:head# 2020-03-11T22:37:43.482 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:43.493 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:be8489b7:::existing_38:head# 2020-03-11T22:37:43.494 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:43.502 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:beb4e5e3:::existing_47:head# 2020-03-11T22:37:43.502 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:43.510 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:bebec877:::existing_74:head# 2020-03-11T22:37:43.510 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:43.518 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:c04347d8:::existing_60:head# 2020-03-11T22:37:43.519 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:43.527 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:c156033e:::existing_100:head# 2020-03-11T22:37:43.527 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:43.535 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:c19f5a8e:::existing_8:head# 2020-03-11T22:37:43.535 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:43.543 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:c3e6ebe4:::existing_17:head# 2020-03-11T22:37:43.544 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:43.552 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:c6c5a7c2:::existing_39:head# 2020-03-11T22:37:43.552 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:43.560 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:cc1d0abf:::existing_95:head# 2020-03-11T22:37:43.560 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:43.569 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:ccb3865b:::existing_73:head# 2020-03-11T22:37:43.569 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:43.577 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:cd84e19e:::existing_40:head# 2020-03-11T22:37:43.577 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:43.585 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:d2cd64ce:::existing_37:head# 2020-03-11T22:37:43.585 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:43.594 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:d41a25c9:::existing_22:head# 2020-03-11T22:37:43.594 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:43.623 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:d7f088da:::existing_91:head# 2020-03-11T22:37:43.624 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:43.635 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:d9cbd1d6:::existing_64:head# 2020-03-11T22:37:43.636 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:43.663 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:dcc4ff82:::existing_98:head# 2020-03-11T22:37:43.663 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:43.677 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:dd677bd3:::existing_35:head# 2020-03-11T22:37:43.677 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:43.685 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:dd6d90e7:::existing_46:head# 2020-03-11T22:37:43.686 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:43.694 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:e0e22077:::existing_21:head# 2020-03-11T22:37:43.694 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:43.702 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:e255b710:::existing_55:head# 2020-03-11T22:37:43.702 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:43.711 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:eff1d5db:::existing_14:head# 2020-03-11T22:37:43.711 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:43.719 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:f1b156bf:::existing_23:head# 2020-03-11T22:37:43.719 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:43.727 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:f5ce4199:::existing_34:head# 2020-03-11T22:37:43.727 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:43.735 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:f6809456:::existing_11:head# 2020-03-11T22:37:43.736 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:43.744 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:f699a2a9:::existing_67:head# 2020-03-11T22:37:43.744 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:43.752 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:f9baac43:::existing_25:head# 2020-03-11T22:37:43.752 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:43.786 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:fb76688e:::existing_51:head# 2020-03-11T22:37:43.786 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:43.832 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:fbb92eec:::existing_69:head# 2020-03-11T22:37:43.832 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:37:43.844 INFO:tasks.workunit.client.0.smithi059.stdout:write_pg epoch 45 info 1.0( v 45'102 lc 0'0 (28'90,45'102] local-lis/les=44/45 n=101 ec=22/22 lis/c=44/22 les/c/f=45/23/0 sis=44 pruub=7.848089054s) 2020-03-11T22:37:43.852 INFO:tasks.workunit.client.0.smithi059.stdout:Import successful 2020-03-11T22:37:44.376 INFO:tasks.workunit.client.0.smithi059.stdout:reviving divergent 1 2020-03-11T22:37:44.377 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:576: TEST_divergent_2: echo 'reviving divergent 1' 2020-03-11T22:37:44.377 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:577: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:37:44.727 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:37:44.728 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 100 4 4 0 0 47947 0 0 11 11 active+recovering+degraded+wait 2020-03-11T22:37:32.500917+0000 40'101 45:125 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:35:29.715580+0000 0'0 2020-03-11T22:35:29.715580+0000 0 2020-03-11T22:37:44.728 INFO:tasks.workunit.client.0.smithi059.stdout: 2020-03-11T22:37:44.728 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:37:44.729 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2020-03-11T22:37:44.740 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:578: TEST_divergent_2: activate_osd td/divergent-priors 1 2020-03-11T22:37:44.740 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:37:44.740 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:37:44.741 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-11T22:37:44.741 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:37:44.741 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:37:44.741 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=cb7f2e42-1934-4145-a3c3-29bdd93f8adb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:37:44.741 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:37:44.741 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:37:44.742 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:37:44.742 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:37:44.742 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:37:44.742 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:37:44.742 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:37:44.742 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:37:44.743 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:37:44.743 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:37:44.743 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:37:44.743 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:37:44.743 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:37:44.743 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:37:44.743 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17931/$cluster-$name.asok' 2020-03-11T22:37:44.744 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' 2020-03-11T22:37:44.744 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:37:44.744 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:37:44.744 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:37:44.744 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:37:44.744 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:37:44.745 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:37:44.745 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:37:44.745 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:37:44.745 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:37:44.745 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-11T22:37:44.745 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=cb7f2e42-1934-4145-a3c3-29bdd93f8adb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:37:44.745 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-03-11T22:37:44.746 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.1 2020-03-11T22:37:44.746 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-11T22:37:44.746 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:37:44.747 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:37:44.747 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:37:44.767 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:37:44.765+0000 7fa24a049ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:44.769 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:37:44.767+0000 7fa24a049ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:44.771 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:37:44.768+0000 7fa24a049ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:45.110 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 1 2020-03-11T22:37:45.110 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:37:45.110 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:37:45.111 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:37:45.111 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:37:45.111 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:37:45.111 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:37:45.111 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:37:45.111 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:37:45.111 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:37:45.289 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:37:45.286+0000 7fa24a049ec0 -1 Falling back to public interface 2020-03-11T22:37:45.475 INFO:tasks.workunit.client.0.smithi059.stdout:osd.1 up in weight 1 up_from 44 up_thru 44 down_at 30 last_clean_interval [10,28) [v2:127.0.0.1:6818/33540,v1:127.0.0.1:6819/33540] [v2:127.0.0.1:6820/33540,v1:127.0.0.1:6821/33540] exists,up 55fb3134-fe56-4d99-9135-8d0c47f417da 2020-03-11T22:37:45.476 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:37:45.476 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:37:45.476 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:37:45.476 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:37:45.476 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:579: TEST_divergent_2: wait_for_osd up 1 2020-03-11T22:37:45.477 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:37:45.477 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:37:45.477 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:37:45.477 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:37:45.477 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:37:45.477 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:37:45.478 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:37:45.478 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:37:45.838 INFO:tasks.workunit.client.0.smithi059.stdout:osd.1 up in weight 1 up_from 44 up_thru 44 down_at 30 last_clean_interval [10,28) [v2:127.0.0.1:6818/33540,v1:127.0.0.1:6819/33540] [v2:127.0.0.1:6820/33540,v1:127.0.0.1:6821/33540] exists,up 55fb3134-fe56-4d99-9135-8d0c47f417da 2020-03-11T22:37:45.839 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:37:45.839 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:37:45.839 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:37:45.839 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:581: TEST_divergent_2: sleep 20 2020-03-11T22:37:46.112 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:37:46.109+0000 7fa24a049ec0 -1 osd.1 45 log_to_monitors {default=true} 2020-03-11T22:38:05.841 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: get_asok_path osd.1 2020-03-11T22:38:05.841 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:38:05.841 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:38:05.841 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:38:05.841 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:38:05.842 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:38:05.842 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17931/ceph-osd.1.asok 2020-03-11T22:38:05.842 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: CEPH_ARGS= 2020-03-11T22:38:05.842 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.17931/ceph-osd.1.asok dump_ops_in_flight 2020-03-11T22:38:05.944 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2020-03-11T22:38:05.945 INFO:tasks.workunit.client.0.smithi059.stdout: "ops": [], 2020-03-11T22:38:05.945 INFO:tasks.workunit.client.0.smithi059.stdout: "num_ops": 0 2020-03-11T22:38:05.945 INFO:tasks.workunit.client.0.smithi059.stdout:} 2020-03-11T22:38:05.952 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:584: TEST_divergent_2: echo 'allowing recovery' 2020-03-11T22:38:05.953 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:585: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:38:05.953 INFO:tasks.workunit.client.0.smithi059.stdout:allowing recovery 2020-03-11T22:38:06.292 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:38:06.293 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 101 0 0 0 0 48430 0 0 12 12 active+clean 2020-03-11T22:37:47.493974+0000 45'102 48:148 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:35:29.715580+0000 0'0 2020-03-11T22:35:29.715580+0000 0 2020-03-11T22:38:06.293 INFO:tasks.workunit.client.0.smithi059.stdout: 2020-03-11T22:38:06.293 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:38:06.294 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2020-03-11T22:38:06.302 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2020-03-11T22:38:06.302 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.0 debug kick_recovery_wq 0 2020-03-11T22:38:06.430 INFO:tasks.workunit.client.0.smithi059.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:38:06.438 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2020-03-11T22:38:06.438 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.1 debug kick_recovery_wq 0 2020-03-11T22:38:06.566 INFO:tasks.workunit.client.0.smithi059.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:38:06.574 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2020-03-11T22:38:06.574 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.2 debug kick_recovery_wq 0 2020-03-11T22:38:06.702 INFO:tasks.workunit.client.0.smithi059.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:38:06.709 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:592: TEST_divergent_2: echo 'reading divergent objects' 2020-03-11T22:38:06.709 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:593: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:38:06.710 INFO:tasks.workunit.client.0.smithi059.stdout:reading divergent objects 2020-03-11T22:38:07.049 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:38:07.050 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 101 0 0 0 0 48430 0 0 12 12 active+clean 2020-03-11T22:37:47.493974+0000 45'102 48:148 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:35:29.715580+0000 0'0 2020-03-11T22:35:29.715580+0000 0 2020-03-11T22:38:07.050 INFO:tasks.workunit.client.0.smithi059.stdout: 2020-03-11T22:38:07.050 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:38:07.051 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2020-03-11T22:38:07.060 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: expr 2 + 2 2020-03-11T22:38:07.061 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: seq 1 4 2020-03-11T22:38:07.061 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:38:07.062 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_1 td/divergent-priors/existing 2020-03-11T22:38:07.091 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:38:07.091 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_2 td/divergent-priors/existing 2020-03-11T22:38:07.120 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:38:07.120 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_3 td/divergent-priors/existing 2020-03-11T22:38:07.149 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:38:07.149 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_4 td/divergent-priors/existing 2020-03-11T22:38:07.179 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: seq 1 2 2020-03-11T22:38:07.179 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2020-03-11T22:38:07.180 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:600: TEST_divergent_2: rados -p test get newobject_1 td/divergent-priors/existing 2020-03-11T22:38:07.206 INFO:tasks.workunit.client.0.smithi059.stderr:error getting test/newobject_1: (2) No such file or directory 2020-03-11T22:38:07.208 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2020-03-11T22:38:07.208 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:600: TEST_divergent_2: rados -p test get newobject_2 td/divergent-priors/existing 2020-03-11T22:38:07.235 INFO:tasks.workunit.client.0.smithi059.stderr:error getting test/newobject_2: (2) No such file or directory 2020-03-11T22:38:07.237 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:602: TEST_divergent_2: rm -f td/divergent-priors/existing 2020-03-11T22:38:07.238 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:604: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2020-03-11T22:38:07.239 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:604: TEST_divergent_2: grep _merge_object_divergent_entries td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:38:07.242 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:37:32.192+0000 7efd3c708700 20 _merge_object_divergent_entries: merging hoid 1:70b3bf12:::existing_4:head entries: 28'105 (28'4) delete 1:70b3bf12:::existing_4:head by client.4540.0:1 2020-03-11T22:35:53.024073+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:38:07.242 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:37:32.192+0000 7efd3c708700 20 _merge_object_divergent_entries: keeping 28'105 (28'4) delete 1:70b3bf12:::existing_4:head by client.4540.0:1 2020-03-11T22:35:53.024073+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:38:07.242 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:37:32.192+0000 7efd3c708700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:38:07.242 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:37:32.192+0000 7efd3c708700 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head prior_version: 28'4 first_divergent_update: 28'105 last_divergent_update: 28'105 2020-03-11T22:38:07.242 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:37:32.192+0000 7efd3c708700 10 _merge_object_divergent_entries: more recent entry found: 40'101 (28'4) modify 1:70b3bf12:::existing_4:head by client.4652.0:1 2020-03-11T22:36:48.283725+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0, already merged 2020-03-11T22:38:07.242 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:37:32.192+0000 7efd3c708700 20 _merge_object_divergent_entries: merging hoid 1:75ac5091:::newobject_2:head entries: 28'101 (0'0) modify 1:75ac5091:::newobject_2:head by client.4527.0:1 2020-03-11T22:35:53.003333+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:38:07.243 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:37:32.192+0000 7efd3c708700 20 _merge_object_divergent_entries: keeping 28'101 (0'0) modify 1:75ac5091:::newobject_2:head by client.4527.0:1 2020-03-11T22:35:53.003333+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:38:07.243 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:37:32.192+0000 7efd3c708700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:38:07.243 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:37:32.192+0000 7efd3c708700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version: 0'0 first_divergent_update: 28'101 last_divergent_update: 28'101 2020-03-11T22:38:07.243 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:37:32.192+0000 7efd3c708700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head has no more recent entries in log 2020-03-11T22:38:07.243 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:37:32.192+0000 7efd3c708700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version or op type indicates creation, deleting 2020-03-11T22:38:07.243 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:37:32.192+0000 7efd3c708700 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 28'106 (28'3) delete 1:8881e6d8:::existing_3:head by client.4541.0:1 2020-03-11T22:35:53.025300+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:38:07.243 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:37:32.192+0000 7efd3c708700 20 _merge_object_divergent_entries: keeping 28'106 (28'3) delete 1:8881e6d8:::existing_3:head by client.4541.0:1 2020-03-11T22:35:53.025300+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:38:07.244 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:37:32.192+0000 7efd3c708700 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2020-03-11T22:38:07.244 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:37:32.192+0000 7efd3c708700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 28'3 first_divergent_update: 28'106 last_divergent_update: 28'106 2020-03-11T22:38:07.244 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:37:32.192+0000 7efd3c708700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2020-03-11T22:38:07.244 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:37:32.192+0000 7efd3c708700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2020-03-11T22:38:07.244 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:37:32.192+0000 7efd3c708700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 28'106 2020-03-11T22:38:07.244 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:37:32.192+0000 7efd3c708700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 28'106 (28'3) delete 1:8881e6d8:::existing_3:head by client.4541.0:1 2020-03-11T22:35:53.025300+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:38:07.245 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:37:32.192+0000 7efd3c708700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2020-03-11T22:38:07.245 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:37:32.192+0000 7efd3c708700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version 28'3 <= info.log_tail 28'90 2020-03-11T22:38:07.245 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:37:32.192+0000 7efd3c708700 20 _merge_object_divergent_entries: merging hoid 1:97db2a0f:::existing_2:head entries: 28'104 (28'2) modify 1:97db2a0f:::existing_2:head by client.4536.0:1 2020-03-11T22:35:53.020217+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:38:07.245 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:37:32.192+0000 7efd3c708700 20 _merge_object_divergent_entries: keeping 28'104 (28'2) modify 1:97db2a0f:::existing_2:head by client.4536.0:1 2020-03-11T22:35:53.020217+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:38:07.245 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:37:32.192+0000 7efd3c708700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:38:07.245 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:37:32.192+0000 7efd3c708700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version: 28'2 first_divergent_update: 28'104 last_divergent_update: 28'104 2020-03-11T22:38:07.245 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:37:32.192+0000 7efd3c708700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head has no more recent entries in log 2020-03-11T22:38:07.246 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:37:32.192+0000 7efd3c708700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head must be rolled back or recovered, attempting to rollback 2020-03-11T22:38:07.246 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:37:32.192+0000 7efd3c708700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head olog_can_rollback_to: 28'106 2020-03-11T22:38:07.246 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:37:32.192+0000 7efd3c708700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot rollback 28'104 (28'2) modify 1:97db2a0f:::existing_2:head by client.4536.0:1 2020-03-11T22:35:53.020217+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:38:07.246 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:37:32.192+0000 7efd3c708700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot roll back, removing and adding to missing 2020-03-11T22:38:07.246 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:37:32.192+0000 7efd3c708700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version 28'2 <= info.log_tail 28'90 2020-03-11T22:38:07.246 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:37:32.192+0000 7efd3c708700 20 _merge_object_divergent_entries: merging hoid 1:d858c035:::existing_1:head entries: 28'102 (28'1) modify 1:d858c035:::existing_1:head by client.4529.0:1 2020-03-11T22:35:53.007053+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:38:07.246 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:37:32.192+0000 7efd3c708700 20 _merge_object_divergent_entries: keeping 28'102 (28'1) modify 1:d858c035:::existing_1:head by client.4529.0:1 2020-03-11T22:35:53.007053+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:38:07.247 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:37:32.192+0000 7efd3c708700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:38:07.247 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:37:32.192+0000 7efd3c708700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version: 28'1 first_divergent_update: 28'102 last_divergent_update: 28'102 2020-03-11T22:38:07.248 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:37:32.192+0000 7efd3c708700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head has no more recent entries in log 2020-03-11T22:38:07.248 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:37:32.192+0000 7efd3c708700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head must be rolled back or recovered, attempting to rollback 2020-03-11T22:38:07.248 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:37:32.192+0000 7efd3c708700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head olog_can_rollback_to: 28'106 2020-03-11T22:38:07.248 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:37:32.192+0000 7efd3c708700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot rollback 28'102 (28'1) modify 1:d858c035:::existing_1:head by client.4529.0:1 2020-03-11T22:35:53.007053+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:38:07.248 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:37:32.192+0000 7efd3c708700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot roll back, removing and adding to missing 2020-03-11T22:38:07.249 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:37:32.192+0000 7efd3c708700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version 28'1 <= info.log_tail 28'90 2020-03-11T22:38:07.249 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:37:32.192+0000 7efd3c708700 20 _merge_object_divergent_entries: merging hoid 1:da59f6e7:::newobject_1:head entries: 28'103 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4533.0:1 2020-03-11T22:35:53.017921+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:38:07.249 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:37:32.192+0000 7efd3c708700 20 _merge_object_divergent_entries: keeping 28'103 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4533.0:1 2020-03-11T22:35:53.017921+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:38:07.249 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:37:32.192+0000 7efd3c708700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:38:07.249 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:37:32.192+0000 7efd3c708700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version: 0'0 first_divergent_update: 28'103 last_divergent_update: 28'103 2020-03-11T22:38:07.249 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:37:32.192+0000 7efd3c708700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head has no more recent entries in log 2020-03-11T22:38:07.250 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:37:32.192+0000 7efd3c708700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version or op type indicates creation, deleting 2020-03-11T22:38:07.250 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:606: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2020-03-11T22:38:07.250 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:606: TEST_divergent_2: grep -q '_merge_object_divergent_entries: more recent entry found:' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:38:07.250 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:612: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2020-03-11T22:38:07.250 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:612: TEST_divergent_2: grep -q '_merge_object_divergent_entries.*prior_version or op type indicates creation' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:38:07.251 INFO:tasks.workunit.client.0.smithi059.stdout:success 2020-03-11T22:38:07.251 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:617: TEST_divergent_2: echo success 2020-03-11T22:38:07.251 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:619: TEST_divergent_2: rm td/divergent-priors/td/divergent-priors/exp.17931.out 2020-03-11T22:38:07.252 INFO:tasks.workunit.client.0.smithi059.stderr:rm: cannot remove 'td/divergent-priors/td/divergent-priors/exp.17931.out': No such file or directory 2020-03-11T22:38:07.252 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:621: TEST_divergent_2: delete_pool test 2020-03-11T22:38:07.252 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test 2020-03-11T22:38:07.252 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2020-03-11T22:38:07.744 INFO:tasks.workunit.client.0.smithi059.stderr:pool 'test' does not exist 2020-03-11T22:38:07.752 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:622: TEST_divergent_2: kill_daemons td/divergent-priors 2020-03-11T22:38:07.753 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:38:07.753 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:38:07.753 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:38:07.753 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:38:07.753 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:38:07.862 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:38:07.862 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2020-03-11T22:38:07.862 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-11T22:38:07.863 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:38:07.863 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-11T22:38:07.863 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:38:07.863 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:38:07.863 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:38:07.863 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:38:07.863 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:38:07.867 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:38:07.867 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:38:07.868 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:38:07.868 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:38:07.869 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:38:07.869 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:38:07.870 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:38:07.871 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:38:07.871 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:38:07.872 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:38:07.872 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:38:07.872 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:38:07.873 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:38:07.873 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-11T22:38:07.885 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:38:07.886 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:38:07.886 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:38:07.886 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17931 2020-03-11T22:38:07.887 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:38:07.888 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:38:07.888 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2020-03-11T22:38:07.888 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2020-03-11T22:38:07.888 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2020-03-11T22:38:07.888 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2020-03-11T22:38:07.888 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-11T22:38:07.889 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:38:07.889 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-11T22:38:07.889 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:38:07.889 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:38:07.889 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:38:07.889 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:38:07.889 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:38:07.890 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:38:07.890 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:38:07.890 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:38:07.891 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:38:07.891 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:38:07.891 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:38:07.892 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:38:07.893 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:38:07.893 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:38:07.893 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:38:07.894 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:38:07.894 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:38:07.895 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:38:07.895 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-11T22:38:07.896 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:38:07.896 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:38:07.897 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:38:07.897 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17931 2020-03-11T22:38:07.899 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:38:07.900 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:38:07.900 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2020-03-11T22:38:07.900 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-11T22:38:07.900 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:38:07.900 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:38:07.900 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.17931 2020-03-11T22:38:07.900 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-11T22:38:07.902 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-03-11T22:38:07.902 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-11T22:38:07.902 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2020-03-11T22:38:07.902 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_3 td/divergent-priors 2020-03-11T22:38:07.902 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:628: TEST_divergent_3: local dir=td/divergent-priors 2020-03-11T22:38:07.902 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:631: TEST_divergent_3: local dummyfile=/etc/fstab 2020-03-11T22:38:07.903 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:632: TEST_divergent_3: local dummyfile2=/etc/resolv.conf 2020-03-11T22:38:07.903 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:634: TEST_divergent_3: local num_osds=3 2020-03-11T22:38:07.903 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: expr 3 - 1 2020-03-11T22:38:07.903 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: seq 0 2 2020-03-11T22:38:07.904 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: local 'osds=0 2020-03-11T22:38:07.904 INFO:tasks.workunit.client.0.smithi059.stderr:1 2020-03-11T22:38:07.905 INFO:tasks.workunit.client.0.smithi059.stderr:2' 2020-03-11T22:38:07.905 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:636: TEST_divergent_3: run_mon td/divergent-priors a 2020-03-11T22:38:07.905 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2020-03-11T22:38:07.905 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-11T22:38:07.905 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-11T22:38:07.905 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-11T22:38:07.906 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2020-03-11T22:38:07.906 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2020-03-11T22:38:08.123 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-11T22:38:08.123 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:38:08.123 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:38:08.123 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:38:08.124 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:38:08.124 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:38:08.124 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17931/$cluster-$name.asok' 2020-03-11T22:38:08.124 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-11T22:38:08.304 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-11T22:38:08.306 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-11T22:38:08.306 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:38:08.306 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:38:08.307 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-11T22:38:08.307 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-11T22:38:08.307 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:38:08.307 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:38:08.307 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:38:08.307 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:38:08.307 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:38:08.308 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:38:08.308 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17931/ceph-mon.a.asok 2020-03-11T22:38:08.308 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:38:08.308 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.17931/ceph-mon.a.asok config get fsid 2020-03-11T22:38:08.400 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-11T22:38:08.400 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:38:08.400 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:38:08.400 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-11T22:38:08.400 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-11T22:38:08.400 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:38:08.401 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:38:08.401 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:38:08.401 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:38:08.401 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:38:08.401 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:38:08.401 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17931/ceph-mon.a.asok 2020-03-11T22:38:08.402 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:38:08.402 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.17931/ceph-mon.a.asok config get mon_host 2020-03-11T22:38:08.491 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:637: TEST_divergent_3: run_mgr td/divergent-priors x 2020-03-11T22:38:08.491 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/divergent-priors 2020-03-11T22:38:08.492 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-11T22:38:08.492 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-11T22:38:08.492 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-11T22:38:08.492 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/divergent-priors/x 2020-03-11T22:38:08.492 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-11T22:38:09.421 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-11T22:38:09.422 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:38:09.422 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:38:09.422 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:38:09.422 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:38:09.422 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:38:09.422 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17931/$cluster-$name.asok' 2020-03-11T22:38:09.423 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:38:09.423 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:38:09.451 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2020-03-11T22:38:09.451 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 0 2020-03-11T22:38:09.452 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:38:09.452 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:38:09.452 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-11T22:38:09.452 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:38:09.452 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/0 2020-03-11T22:38:09.452 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=cb7f2e42-1934-4145-a3c3-29bdd93f8adb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:38:09.452 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:38:09.453 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:38:09.453 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:38:09.453 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-11T22:38:09.453 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-11T22:38:09.453 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:38:09.453 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:38:09.453 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:38:09.454 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:38:09.454 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:38:09.454 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:38:09.454 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:38:09.454 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:38:09.454 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:38:09.454 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17931/$cluster-$name.asok' 2020-03-11T22:38:09.455 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' 2020-03-11T22:38:09.455 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:38:09.455 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:38:09.455 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:38:09.455 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:38:09.455 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:38:09.455 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:38:09.456 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:38:09.456 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:38:09.456 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:38:09.456 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:38:09.456 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/0 2020-03-11T22:38:09.457 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:38:09.457 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=9e248648-07e4-4309-92d7-2283b32a88ea 2020-03-11T22:38:09.457 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 9e248648-07e4-4309-92d7-2283b32a88ea' 2020-03-11T22:38:09.457 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:38:09.458 INFO:tasks.workunit.client.0.smithi059.stdout:add osd0 9e248648-07e4-4309-92d7-2283b32a88ea 2020-03-11T22:38:09.472 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBRaGle4Sv5GxAAWlC76D5TEtp/gAPliQ0AEQ== 2020-03-11T22:38:09.472 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBRaGle4Sv5GxAAWlC76D5TEtp/gAPliQ0AEQ=="}' 2020-03-11T22:38:09.472 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 9e248648-07e4-4309-92d7-2283b32a88ea -i td/divergent-priors/0/new.json 2020-03-11T22:38:09.822 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:38:09.831 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/0/new.json 2020-03-11T22:38:09.832 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=cb7f2e42-1934-4145-a3c3-29bdd93f8adb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBRaGle4Sv5GxAAWlC76D5TEtp/gAPliQ0AEQ== --osd-uuid 9e248648-07e4-4309-92d7-2283b32a88ea 2020-03-11T22:38:09.862 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:38:09.859+0000 7f04e3372ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:38:09.864 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:38:09.861+0000 7f04e3372ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:38:09.866 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:38:09.863+0000 7f04e3372ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:38:09.866 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:38:09.864+0000 7f04e3372ec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:38:09.867 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:38:09.864+0000 7f04e3372ec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:38:09.867 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:38:09.864+0000 7f04e3372ec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:38:09.895 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:38:09.893+0000 7f04e3372ec0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2020-03-11T22:38:12.956 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/0/keyring 2020-03-11T22:38:12.956 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:38:12.958 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-11T22:38:12.958 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:38:12.958 INFO:tasks.workunit.client.0.smithi059.stdout:adding osd0 key to auth repository 2020-03-11T22:38:13.219 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-11T22:38:13.220 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:38:13.220 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=cb7f2e42-1934-4145-a3c3-29bdd93f8adb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:38:13.220 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:38:13.220 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:38:13.220 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.0 2020-03-11T22:38:13.247 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:38:13.244+0000 7f62035b0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:38:13.249 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:38:13.246+0000 7f62035b0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:38:13.250 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:38:13.248+0000 7f62035b0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:38:13.587 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-11T22:38:13.587 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:38:13.588 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:38:13.588 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:38:13.588 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:38:13.588 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:38:13.588 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:38:13.588 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:38:13.588 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:38:13.589 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:38:13.767 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:38:13.764+0000 7f62035b0ec0 -1 Falling back to public interface 2020-03-11T22:38:13.957 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:38:14.537 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:38:14.534+0000 7f62035b0ec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-11T22:38:14.958 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:38:14.959 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:38:14.959 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:38:14.959 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:38:14.959 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:38:14.959 INFO:tasks.workunit.client.0.smithi059.stdout:1 2020-03-11T22:38:15.355 INFO:tasks.workunit.client.0.smithi059.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/35035,v1:127.0.0.1:6803/35035] [v2:127.0.0.1:6804/35035,v1:127.0.0.1:6805/35035] exists,up 9e248648-07e4-4309-92d7-2283b32a88ea 2020-03-11T22:38:15.355 INFO:tasks.workunit.client.0.smithi059.stdout:add osd1 98cb8180-5961-424c-a790-9139892a46ae 2020-03-11T22:38:15.355 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:38:15.356 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:38:15.356 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:38:15.356 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2020-03-11T22:38:15.356 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 1 2020-03-11T22:38:15.356 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:38:15.356 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:38:15.357 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-11T22:38:15.357 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:38:15.357 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:38:15.357 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=cb7f2e42-1934-4145-a3c3-29bdd93f8adb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:38:15.357 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:38:15.357 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:38:15.357 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:38:15.358 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:38:15.358 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:38:15.358 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:38:15.358 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:38:15.358 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:38:15.358 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:38:15.358 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:38:15.359 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:38:15.359 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:38:15.359 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:38:15.359 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:38:15.359 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17931/$cluster-$name.asok' 2020-03-11T22:38:15.359 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' 2020-03-11T22:38:15.359 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:38:15.360 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:38:15.360 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:38:15.360 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:38:15.360 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:38:15.360 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:38:15.360 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:38:15.360 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:38:15.360 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:38:15.361 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:38:15.361 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:38:15.361 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:38:15.361 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=98cb8180-5961-424c-a790-9139892a46ae 2020-03-11T22:38:15.361 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 98cb8180-5961-424c-a790-9139892a46ae' 2020-03-11T22:38:15.362 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:38:15.374 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBXaGlemhAoFhAAHf9OiSy0OFLLI95vf/t0ow== 2020-03-11T22:38:15.374 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBXaGlemhAoFhAAHf9OiSy0OFLLI95vf/t0ow=="}' 2020-03-11T22:38:15.374 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 98cb8180-5961-424c-a790-9139892a46ae -i td/divergent-priors/1/new.json 2020-03-11T22:38:15.901 INFO:tasks.workunit.client.0.smithi059.stdout:1 2020-03-11T22:38:15.911 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/1/new.json 2020-03-11T22:38:15.912 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=cb7f2e42-1934-4145-a3c3-29bdd93f8adb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBXaGlemhAoFhAAHf9OiSy0OFLLI95vf/t0ow== --osd-uuid 98cb8180-5961-424c-a790-9139892a46ae 2020-03-11T22:38:15.942 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:38:15.939+0000 7fef9c000ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:38:15.944 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:38:15.941+0000 7fef9c000ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:38:15.945 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:38:15.943+0000 7fef9c000ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:38:15.946 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:38:15.943+0000 7fef9c000ec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:38:15.946 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:38:15.943+0000 7fef9c000ec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:38:15.946 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:38:15.943+0000 7fef9c000ec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:38:16.002 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:38:16.000+0000 7fef9c000ec0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2020-03-11T22:38:19.127 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/1/keyring 2020-03-11T22:38:19.128 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:38:19.129 INFO:tasks.workunit.client.0.smithi059.stdout:adding osd1 key to auth repository 2020-03-11T22:38:19.129 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-03-11T22:38:19.129 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:38:19.614 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.1 2020-03-11T22:38:19.614 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-03-11T22:38:19.615 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:38:19.615 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=cb7f2e42-1934-4145-a3c3-29bdd93f8adb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:38:19.615 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:38:19.615 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:38:19.642 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:38:19.640+0000 7f6cd84ebec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:38:19.644 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:38:19.642+0000 7f6cd84ebec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:38:19.645 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:38:19.643+0000 7f6cd84ebec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:38:19.983 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-03-11T22:38:19.983 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:38:19.983 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:38:19.983 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:38:19.983 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:38:19.983 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:38:19.984 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:38:19.984 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:38:19.984 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:38:19.984 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:38:20.163 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:38:20.160+0000 7f6cd84ebec0 -1 Falling back to public interface 2020-03-11T22:38:20.406 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:38:20.893 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:38:20.890+0000 7f6cd84ebec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-11T22:38:21.407 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:38:21.408 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:38:21.408 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:38:21.408 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:38:21.408 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:38:21.408 INFO:tasks.workunit.client.0.smithi059.stdout:1 2020-03-11T22:38:21.790 INFO:tasks.workunit.client.0.smithi059.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/35398,v1:127.0.0.1:6811/35398] [v2:127.0.0.1:6812/35398,v1:127.0.0.1:6813/35398] exists,up 98cb8180-5961-424c-a790-9139892a46ae 2020-03-11T22:38:21.790 INFO:tasks.workunit.client.0.smithi059.stdout:add osd2 18b16e96-77ce-45af-9ce0-23852cb42c25 2020-03-11T22:38:21.790 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:38:21.791 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:38:21.791 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:38:21.791 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2020-03-11T22:38:21.791 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 2 2020-03-11T22:38:21.791 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:38:21.792 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:38:21.792 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-03-11T22:38:21.792 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:38:21.792 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/2 2020-03-11T22:38:21.792 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=cb7f2e42-1934-4145-a3c3-29bdd93f8adb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:38:21.792 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:38:21.793 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:38:21.793 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:38:21.793 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-11T22:38:21.793 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-11T22:38:21.793 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:38:21.793 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:38:21.794 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:38:21.794 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:38:21.794 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:38:21.794 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:38:21.794 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:38:21.795 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:38:21.795 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:38:21.795 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17931/$cluster-$name.asok' 2020-03-11T22:38:21.795 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' 2020-03-11T22:38:21.795 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:38:21.795 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:38:21.795 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:38:21.796 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:38:21.796 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:38:21.796 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:38:21.796 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:38:21.796 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:38:21.796 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:38:21.797 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:38:21.797 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/2 2020-03-11T22:38:21.797 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:38:21.797 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=18b16e96-77ce-45af-9ce0-23852cb42c25 2020-03-11T22:38:21.797 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 18b16e96-77ce-45af-9ce0-23852cb42c25' 2020-03-11T22:38:21.797 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:38:21.809 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBdaGleAcYMMBAAuSe9ySB0K70kj/bBQWXqkg== 2020-03-11T22:38:21.809 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBdaGleAcYMMBAAuSe9ySB0K70kj/bBQWXqkg=="}' 2020-03-11T22:38:21.809 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 18b16e96-77ce-45af-9ce0-23852cb42c25 -i td/divergent-priors/2/new.json 2020-03-11T22:38:22.313 INFO:tasks.workunit.client.0.smithi059.stdout:2 2020-03-11T22:38:22.322 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/2/new.json 2020-03-11T22:38:22.323 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=cb7f2e42-1934-4145-a3c3-29bdd93f8adb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBdaGleAcYMMBAAuSe9ySB0K70kj/bBQWXqkg== --osd-uuid 18b16e96-77ce-45af-9ce0-23852cb42c25 2020-03-11T22:38:22.353 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:38:22.350+0000 7f1c07154ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:38:22.408 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:38:22.405+0000 7f1c07154ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:38:22.409 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:38:22.407+0000 7f1c07154ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:38:22.410 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:38:22.407+0000 7f1c07154ec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:38:22.410 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:38:22.407+0000 7f1c07154ec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:38:22.410 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:38:22.407+0000 7f1c07154ec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:38:22.474 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:38:22.472+0000 7f1c07154ec0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2020-03-11T22:38:25.608 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/2/keyring 2020-03-11T22:38:25.609 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:38:25.610 INFO:tasks.workunit.client.0.smithi059.stdout:adding osd2 key to auth repository 2020-03-11T22:38:25.610 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-03-11T22:38:25.610 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:38:26.099 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.2 2020-03-11T22:38:26.100 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-03-11T22:38:26.100 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=cb7f2e42-1934-4145-a3c3-29bdd93f8adb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:38:26.100 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:38:26.100 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:38:26.100 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:38:26.127 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:38:26.123+0000 7f5c4cd8fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:38:26.129 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:38:26.125+0000 7f5c4cd8fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:38:26.131 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:38:26.127+0000 7f5c4cd8fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:38:26.462 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-03-11T22:38:26.463 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:38:26.463 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:38:26.463 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:38:26.463 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:38:26.463 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:38:26.463 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:38:26.463 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:38:26.464 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:38:26.464 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:38:26.648 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:38:26.644+0000 7f5c4cd8fec0 -1 Falling back to public interface 2020-03-11T22:38:26.859 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:38:27.349 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:38:27.346+0000 7f5c4cd8fec0 -1 osd.2 0 log_to_monitors {default=true} 2020-03-11T22:38:27.861 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:38:27.861 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:38:27.861 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:38:27.861 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:38:27.861 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:38:27.861 INFO:tasks.workunit.client.0.smithi059.stdout:1 2020-03-11T22:38:28.236 INFO:tasks.workunit.client.0.smithi059.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/35753,v1:127.0.0.1:6819/35753] [v2:127.0.0.1:6820/35753,v1:127.0.0.1:6821/35753] exists,up 18b16e96-77ce-45af-9ce0-23852cb42c25 2020-03-11T22:38:28.237 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:38:28.237 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:38:28.237 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:38:28.237 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:643: TEST_divergent_3: ceph osd set noout 2020-03-11T22:38:28.934 INFO:tasks.workunit.client.0.smithi059.stderr:noout is set 2020-03-11T22:38:28.942 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:644: TEST_divergent_3: ceph osd set noin 2020-03-11T22:38:29.687 INFO:tasks.workunit.client.0.smithi059.stderr:noin is set 2020-03-11T22:38:29.695 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:645: TEST_divergent_3: ceph osd set nodown 2020-03-11T22:38:30.472 INFO:tasks.workunit.client.0.smithi059.stderr:nodown is set 2020-03-11T22:38:30.479 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:646: TEST_divergent_3: create_pool test 1 1 2020-03-11T22:38:30.480 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test 1 1 2020-03-11T22:38:31.207 INFO:tasks.workunit.client.0.smithi059.stderr:pool 'test' already exists 2020-03-11T22:38:31.215 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:38:32.216 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:647: TEST_divergent_3: ceph osd pool set test size 3 2020-03-11T22:38:32.983 INFO:tasks.workunit.client.0.smithi059.stderr:set pool 1 size to 3 2020-03-11T22:38:32.992 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:648: TEST_divergent_3: ceph osd pool set test min_size 2 2020-03-11T22:38:33.715 INFO:tasks.workunit.client.0.smithi059.stderr:set pool 1 min_size to 2 2020-03-11T22:38:33.723 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:651: TEST_divergent_3: ceph osd pool set test pg_autoscale_mode on 2020-03-11T22:38:34.346 INFO:tasks.workunit.client.0.smithi059.stderr:set pool 1 pg_autoscale_mode to on 2020-03-11T22:38:34.354 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:653: TEST_divergent_3: flush_pg_stats 2020-03-11T22:38:34.354 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:38:34.354 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:38:34.721 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:38:34.722 INFO:tasks.workunit.client.0.smithi059.stderr:1 2020-03-11T22:38:34.722 INFO:tasks.workunit.client.0.smithi059.stderr:2' 2020-03-11T22:38:34.722 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:38:34.722 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:38:34.723 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:38:34.895 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836487 2020-03-11T22:38:34.896 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836487' 2020-03-11T22:38:34.896 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:38:34.896 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:38:35.032 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672965 2020-03-11T22:38:35.032 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836487 1-42949672965' 2020-03-11T22:38:35.032 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:38:35.033 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:38:35.169 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509444 2020-03-11T22:38:35.170 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836487 1-42949672965 2-64424509444' 2020-03-11T22:38:35.170 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:38:35.170 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836487 2020-03-11T22:38:35.170 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:38:35.172 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:38:35.172 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836487 2020-03-11T22:38:35.172 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:38:35.173 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836487 2020-03-11T22:38:35.173 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2020-03-11T22:38:35.173 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:38:35.173 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.0 seq 21474836487 2020-03-11T22:38:35.535 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836487 -lt 21474836487 2020-03-11T22:38:35.535 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:38:35.536 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672965 2020-03-11T22:38:35.536 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:38:35.537 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:38:35.537 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672965 2020-03-11T22:38:35.537 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:38:35.539 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672965 2020-03-11T22:38:35.539 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2020-03-11T22:38:35.539 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:38:35.539 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.1 seq 42949672965 2020-03-11T22:38:35.915 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672963 -lt 42949672965 2020-03-11T22:38:35.916 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:38:36.917 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:38:36.917 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:38:37.292 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672965 -lt 42949672965 2020-03-11T22:38:37.292 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:38:37.292 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509444 2020-03-11T22:38:37.293 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:38:37.293 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:38:37.294 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509444 2020-03-11T22:38:37.294 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:38:37.295 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509444 2020-03-11T22:38:37.295 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2020-03-11T22:38:37.295 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.2 seq 64424509444 2020-03-11T22:38:37.295 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:38:37.676 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509444 -lt 64424509444 2020-03-11T22:38:37.676 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:654: TEST_divergent_3: wait_for_clean 2020-03-11T22:38:37.676 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:38:37.676 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:38:37.677 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:38:37.677 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:38:37.677 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:38:37.677 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:38:37.677 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:38:37.677 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:38:37.677 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:38:37.678 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:38:37.773 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:38:37.773 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:38:37.773 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:38:37.773 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:38:37.773 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:38:38.138 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:38:38.139 INFO:tasks.workunit.client.0.smithi059.stderr:1 2020-03-11T22:38:38.139 INFO:tasks.workunit.client.0.smithi059.stderr:2' 2020-03-11T22:38:38.139 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:38:38.139 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:38:38.139 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:38:38.297 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836490 2020-03-11T22:38:38.297 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836490' 2020-03-11T22:38:38.298 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:38:38.298 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:38:38.434 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672968 2020-03-11T22:38:38.434 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836490 1-42949672968' 2020-03-11T22:38:38.434 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:38:38.435 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:38:38.571 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509446 2020-03-11T22:38:38.571 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836490 1-42949672968 2-64424509446' 2020-03-11T22:38:38.571 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:38:38.572 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836490 2020-03-11T22:38:38.572 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:38:38.572 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:38:38.573 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836490 2020-03-11T22:38:38.573 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:38:38.574 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836490 2020-03-11T22:38:38.574 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2020-03-11T22:38:38.574 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:38:38.574 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.0 seq 21474836490 2020-03-11T22:38:38.945 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836488 -lt 21474836490 2020-03-11T22:38:38.945 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:38:39.946 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:38:39.947 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:38:40.319 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836490 -lt 21474836490 2020-03-11T22:38:40.319 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:38:40.320 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672968 2020-03-11T22:38:40.320 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:38:40.320 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:38:40.321 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672968 2020-03-11T22:38:40.321 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:38:40.322 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672968 2020-03-11T22:38:40.322 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2020-03-11T22:38:40.322 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:38:40.323 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.1 seq 42949672968 2020-03-11T22:38:40.766 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672968 -lt 42949672968 2020-03-11T22:38:40.766 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:38:40.767 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509446 2020-03-11T22:38:40.767 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:38:40.767 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:38:40.768 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509446 2020-03-11T22:38:40.768 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:38:40.769 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509446 2020-03-11T22:38:40.769 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2020-03-11T22:38:40.770 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:38:40.770 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.2 seq 64424509446 2020-03-11T22:38:41.132 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509447 -lt 64424509446 2020-03-11T22:38:41.133 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:38:41.133 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:38:41.133 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:38:41.610 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:38:41.610 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:38:41.611 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:38:41.611 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:38:41.611 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:38:41.611 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:38:41.611 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:38:41.611 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:38:41.960 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:38:41.961 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:38:41.961 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:38:41.961 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:38:42.438 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:38:42.438 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:38:42.438 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:38:42.438 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: ceph pg dump pgs --format=json 2020-03-11T22:38:42.439 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: jq '.pg_stats[0].up_primary' 2020-03-11T22:38:42.784 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2020-03-11T22:38:42.792 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: local divergent=1 2020-03-11T22:38:42.792 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:658: TEST_divergent_3: echo 'primary and soon to be divergent is 1' 2020-03-11T22:38:42.793 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:659: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:38:42.793 INFO:tasks.workunit.client.0.smithi059.stdout:primary and soon to be divergent is 1 2020-03-11T22:38:43.133 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:38:43.134 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:38:31.478899+0000 0'0 30:12 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:38:43.134 INFO:tasks.workunit.client.0.smithi059.stdout: 2020-03-11T22:38:43.134 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:38:43.135 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2020-03-11T22:38:43.144 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:660: TEST_divergent_3: local non_divergent= 2020-03-11T22:38:43.144 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:661: TEST_divergent_3: for i in $osds 2020-03-11T22:38:43.144 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: '[' 0 = 1 ']' 2020-03-11T22:38:43.145 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:666: TEST_divergent_3: non_divergent=' 0' 2020-03-11T22:38:43.145 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:661: TEST_divergent_3: for i in $osds 2020-03-11T22:38:43.145 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: '[' 1 = 1 ']' 2020-03-11T22:38:43.145 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:664: TEST_divergent_3: continue 2020-03-11T22:38:43.145 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:661: TEST_divergent_3: for i in $osds 2020-03-11T22:38:43.145 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: '[' 2 = 1 ']' 2020-03-11T22:38:43.145 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:666: TEST_divergent_3: non_divergent=' 0 2' 2020-03-11T22:38:43.146 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:669: TEST_divergent_3: echo 'writing initial objects' 2020-03-11T22:38:43.146 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: seq 1 100 2020-03-11T22:38:43.146 INFO:tasks.workunit.client.0.smithi059.stdout:writing initial objects 2020-03-11T22:38:43.146 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:43.146 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_1 /etc/fstab 2020-03-11T22:38:43.250 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:43.250 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_2 /etc/fstab 2020-03-11T22:38:43.331 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:43.331 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_3 /etc/fstab 2020-03-11T22:38:43.372 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:43.372 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_4 /etc/fstab 2020-03-11T22:38:43.479 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:43.479 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_5 /etc/fstab 2020-03-11T22:38:43.531 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:43.531 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_6 /etc/fstab 2020-03-11T22:38:43.606 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:43.606 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_7 /etc/fstab 2020-03-11T22:38:43.655 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:43.655 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_8 /etc/fstab 2020-03-11T22:38:43.706 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:43.706 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_9 /etc/fstab 2020-03-11T22:38:43.756 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:43.756 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_10 /etc/fstab 2020-03-11T22:38:43.805 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:43.806 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_11 /etc/fstab 2020-03-11T22:38:43.876 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:43.876 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_12 /etc/fstab 2020-03-11T22:38:43.939 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:43.940 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_13 /etc/fstab 2020-03-11T22:38:44.018 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:44.019 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_14 /etc/fstab 2020-03-11T22:38:44.102 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:44.102 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_15 /etc/fstab 2020-03-11T22:38:44.172 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:44.173 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_16 /etc/fstab 2020-03-11T22:38:44.223 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:44.223 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_17 /etc/fstab 2020-03-11T22:38:44.302 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:44.302 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_18 /etc/fstab 2020-03-11T22:38:44.347 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:44.347 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_19 /etc/fstab 2020-03-11T22:38:44.397 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:44.398 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_20 /etc/fstab 2020-03-11T22:38:44.456 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:44.456 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_21 /etc/fstab 2020-03-11T22:38:44.506 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:44.506 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_22 /etc/fstab 2020-03-11T22:38:44.558 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:44.558 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_23 /etc/fstab 2020-03-11T22:38:44.598 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:44.598 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_24 /etc/fstab 2020-03-11T22:38:44.661 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:44.661 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_25 /etc/fstab 2020-03-11T22:38:44.770 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:44.770 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_26 /etc/fstab 2020-03-11T22:38:44.861 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:44.862 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_27 /etc/fstab 2020-03-11T22:38:44.914 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:44.914 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_28 /etc/fstab 2020-03-11T22:38:45.045 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:45.046 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_29 /etc/fstab 2020-03-11T22:38:45.098 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:45.098 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_30 /etc/fstab 2020-03-11T22:38:45.148 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:45.148 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_31 /etc/fstab 2020-03-11T22:38:45.198 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:45.198 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_32 /etc/fstab 2020-03-11T22:38:45.248 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:45.248 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_33 /etc/fstab 2020-03-11T22:38:45.306 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:45.306 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_34 /etc/fstab 2020-03-11T22:38:45.356 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:45.356 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_35 /etc/fstab 2020-03-11T22:38:45.445 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:45.446 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_36 /etc/fstab 2020-03-11T22:38:45.498 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:45.498 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_37 /etc/fstab 2020-03-11T22:38:45.654 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:45.654 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_38 /etc/fstab 2020-03-11T22:38:45.698 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:45.698 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_39 /etc/fstab 2020-03-11T22:38:45.747 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:45.748 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_40 /etc/fstab 2020-03-11T22:38:45.838 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:45.838 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_41 /etc/fstab 2020-03-11T22:38:45.889 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:45.889 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_42 /etc/fstab 2020-03-11T22:38:45.931 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:45.931 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_43 /etc/fstab 2020-03-11T22:38:45.989 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:45.990 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_44 /etc/fstab 2020-03-11T22:38:46.039 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:46.040 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_45 /etc/fstab 2020-03-11T22:38:46.090 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:46.090 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_46 /etc/fstab 2020-03-11T22:38:46.165 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:46.165 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_47 /etc/fstab 2020-03-11T22:38:46.215 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:46.215 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_48 /etc/fstab 2020-03-11T22:38:46.305 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:46.305 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_49 /etc/fstab 2020-03-11T22:38:46.397 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:46.397 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_50 /etc/fstab 2020-03-11T22:38:46.456 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:46.457 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_51 /etc/fstab 2020-03-11T22:38:46.547 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:46.548 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_52 /etc/fstab 2020-03-11T22:38:46.607 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:46.607 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_53 /etc/fstab 2020-03-11T22:38:46.656 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:46.656 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_54 /etc/fstab 2020-03-11T22:38:46.731 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:46.732 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_55 /etc/fstab 2020-03-11T22:38:46.781 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:46.781 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_56 /etc/fstab 2020-03-11T22:38:46.832 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:46.832 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_57 /etc/fstab 2020-03-11T22:38:46.881 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:46.882 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_58 /etc/fstab 2020-03-11T22:38:46.932 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:46.932 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_59 /etc/fstab 2020-03-11T22:38:47.041 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:47.041 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_60 /etc/fstab 2020-03-11T22:38:47.173 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:47.173 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_61 /etc/fstab 2020-03-11T22:38:47.298 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:47.299 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_62 /etc/fstab 2020-03-11T22:38:47.357 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:47.357 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_63 /etc/fstab 2020-03-11T22:38:47.449 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:47.449 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_64 /etc/fstab 2020-03-11T22:38:47.507 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:47.507 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_65 /etc/fstab 2020-03-11T22:38:47.581 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:47.582 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_66 /etc/fstab 2020-03-11T22:38:47.674 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:47.674 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_67 /etc/fstab 2020-03-11T22:38:47.732 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:47.732 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_68 /etc/fstab 2020-03-11T22:38:47.782 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:47.782 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_69 /etc/fstab 2020-03-11T22:38:47.865 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:47.865 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_70 /etc/fstab 2020-03-11T22:38:47.915 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:47.915 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_71 /etc/fstab 2020-03-11T22:38:48.007 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:48.007 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_72 /etc/fstab 2020-03-11T22:38:48.112 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:48.112 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_73 /etc/fstab 2020-03-11T22:38:48.208 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:48.208 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_74 /etc/fstab 2020-03-11T22:38:48.266 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:48.266 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_75 /etc/fstab 2020-03-11T22:38:48.358 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:48.359 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_76 /etc/fstab 2020-03-11T22:38:48.483 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:48.484 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_77 /etc/fstab 2020-03-11T22:38:48.549 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:48.549 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_78 /etc/fstab 2020-03-11T22:38:48.598 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:48.598 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_79 /etc/fstab 2020-03-11T22:38:48.648 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:48.649 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_80 /etc/fstab 2020-03-11T22:38:48.707 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:48.707 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_81 /etc/fstab 2020-03-11T22:38:48.756 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:48.757 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_82 /etc/fstab 2020-03-11T22:38:48.807 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:48.807 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_83 /etc/fstab 2020-03-11T22:38:48.909 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:48.910 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_84 /etc/fstab 2020-03-11T22:38:49.077 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:49.077 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_85 /etc/fstab 2020-03-11T22:38:49.165 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:49.165 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_86 /etc/fstab 2020-03-11T22:38:49.294 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:49.294 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_87 /etc/fstab 2020-03-11T22:38:49.348 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:49.349 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_88 /etc/fstab 2020-03-11T22:38:49.398 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:49.398 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_89 /etc/fstab 2020-03-11T22:38:49.449 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:49.449 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_90 /etc/fstab 2020-03-11T22:38:49.498 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:49.499 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_91 /etc/fstab 2020-03-11T22:38:49.589 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:49.590 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_92 /etc/fstab 2020-03-11T22:38:49.640 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:49.641 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_93 /etc/fstab 2020-03-11T22:38:49.690 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:49.690 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_94 /etc/fstab 2020-03-11T22:38:49.741 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:49.741 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_95 /etc/fstab 2020-03-11T22:38:49.853 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:49.853 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_96 /etc/fstab 2020-03-11T22:38:49.953 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:49.953 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_97 /etc/fstab 2020-03-11T22:38:50.007 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:50.007 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_98 /etc/fstab 2020-03-11T22:38:50.137 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:50.137 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_99 /etc/fstab 2020-03-11T22:38:50.191 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:38:50.191 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_100 /etc/fstab 2020-03-11T22:38:50.241 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:38:50.241 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: wait_for_clean 2020-03-11T22:38:50.241 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:38:50.241 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:38:50.241 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:38:50.241 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:38:50.242 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:38:50.242 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:38:50.243 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:38:50.243 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:38:50.243 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:38:50.243 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:38:50.301 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:38:50.301 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:38:50.301 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:38:50.301 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:38:50.301 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:38:50.667 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:38:50.667 INFO:tasks.workunit.client.0.smithi059.stderr:1 2020-03-11T22:38:50.667 INFO:tasks.workunit.client.0.smithi059.stderr:2' 2020-03-11T22:38:50.668 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:38:50.668 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:38:50.668 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:38:50.804 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836494 2020-03-11T22:38:50.805 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836494' 2020-03-11T22:38:50.805 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:38:50.805 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:38:50.940 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672972 2020-03-11T22:38:50.940 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836494 1-42949672972' 2020-03-11T22:38:50.941 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:38:50.941 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:38:51.078 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509451 2020-03-11T22:38:51.078 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836494 1-42949672972 2-64424509451' 2020-03-11T22:38:51.079 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:38:51.079 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836494 2020-03-11T22:38:51.079 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:38:51.079 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:38:51.080 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836494 2020-03-11T22:38:51.080 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:38:51.081 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836494 2020-03-11T22:38:51.081 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836494' 2020-03-11T22:38:51.081 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:38:51.081 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.0 seq 21474836494 2020-03-11T22:38:51.442 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836495 -lt 21474836494 2020-03-11T22:38:51.442 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:38:51.442 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672972 2020-03-11T22:38:51.443 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:38:51.443 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:38:51.444 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672972 2020-03-11T22:38:51.444 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:38:51.445 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672972 2020-03-11T22:38:51.445 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2020-03-11T22:38:51.445 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.1 seq 42949672972 2020-03-11T22:38:51.445 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:38:51.808 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672970 -lt 42949672972 2020-03-11T22:38:51.809 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:38:52.810 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:38:52.810 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:38:53.184 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672973 -lt 42949672972 2020-03-11T22:38:53.184 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:38:53.185 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509451 2020-03-11T22:38:53.185 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:38:53.185 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:38:53.186 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509451 2020-03-11T22:38:53.186 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:38:53.187 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509451 2020-03-11T22:38:53.187 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2020-03-11T22:38:53.187 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:38:53.188 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.2 seq 64424509451 2020-03-11T22:38:53.559 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509451 -lt 64424509451 2020-03-11T22:38:53.559 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:38:53.560 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:38:53.560 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:38:54.030 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:38:54.030 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:38:54.031 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:38:54.031 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:38:54.031 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:38:54.031 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:38:54.031 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:38:54.031 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:38:54.379 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:38:54.380 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:38:54.380 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:38:54.380 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:38:54.850 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:38:54.850 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:38:54.851 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:38:54.851 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: get_pg test existing_1 2020-03-11T22:38:54.851 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1030: get_pg: local poolname=test 2020-03-11T22:38:54.851 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local objectname=existing_1 2020-03-11T22:38:54.851 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: ceph --format json osd map test existing_1 2020-03-11T22:38:54.851 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: jq -r .pgid 2020-03-11T22:38:55.218 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: local pgid=1.0 2020-03-11T22:38:55.218 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:681: TEST_divergent_3: echo 'blackholing osds 0 2' 2020-03-11T22:38:55.218 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:682: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:38:55.218 INFO:tasks.workunit.client.0.smithi059.stdout:blackholing osds 0 2 2020-03-11T22:38:55.563 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:38:55.563 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 100 0 0 0 0 48300 0 0 16 16 active+clean 2020-03-11T22:38:31.478899+0000 31'100 31:112 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:38:55.563 INFO:tasks.workunit.client.0.smithi059.stdout: 2020-03-11T22:38:55.563 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:38:55.564 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2020-03-11T22:38:55.572 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:683: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:38:55.573 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: get_asok_path osd.0 2020-03-11T22:38:55.574 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-11T22:38:55.574 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-11T22:38:55.574 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:38:55.574 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:38:55.574 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:38:55.574 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17931/ceph-osd.0.asok 2020-03-11T22:38:55.575 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: CEPH_ARGS= 2020-03-11T22:38:55.575 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.17931/ceph-osd.0.asok config set objectstore_blackhole 1 2020-03-11T22:38:55.665 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2020-03-11T22:38:55.665 INFO:tasks.workunit.client.0.smithi059.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-03-11T22:38:55.665 INFO:tasks.workunit.client.0.smithi059.stdout:} 2020-03-11T22:38:55.672 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:683: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:38:55.673 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: get_asok_path osd.2 2020-03-11T22:38:55.673 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-03-11T22:38:55.673 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-03-11T22:38:55.673 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:38:55.673 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:38:55.674 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:38:55.674 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17931/ceph-osd.2.asok 2020-03-11T22:38:55.674 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: CEPH_ARGS= 2020-03-11T22:38:55.674 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.17931/ceph-osd.2.asok config set objectstore_blackhole 1 2020-03-11T22:38:55.764 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2020-03-11T22:38:55.764 INFO:tasks.workunit.client.0.smithi059.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-03-11T22:38:55.764 INFO:tasks.workunit.client.0.smithi059.stdout:} 2020-03-11T22:38:55.771 INFO:tasks.workunit.client.0.smithi059.stdout:create new divergent objects 2020-03-11T22:38:55.772 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:689: TEST_divergent_3: echo 'create new divergent objects' 2020-03-11T22:38:55.772 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:690: TEST_divergent_3: seq 1 2 2020-03-11T22:38:55.773 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:690: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2020-03-11T22:38:55.773 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:690: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2020-03-11T22:38:55.773 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:692: TEST_divergent_3: rados -p test create newobject_1 2020-03-11T22:38:55.773 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:695: TEST_divergent_3: echo 'writing divergent objects' 2020-03-11T22:38:55.774 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:692: TEST_divergent_3: rados -p test create newobject_2 2020-03-11T22:38:55.774 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:696: TEST_divergent_3: seq 1 2 2020-03-11T22:38:55.774 INFO:tasks.workunit.client.0.smithi059.stdout:writing divergent objects 2020-03-11T22:38:55.775 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:696: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_WRITE) 2020-03-11T22:38:55.775 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:696: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_WRITE) 2020-03-11T22:38:55.775 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:698: TEST_divergent_3: rados -p test put existing_1 /etc/resolv.conf 2020-03-11T22:38:55.775 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:701: TEST_divergent_3: echo 'remove divergent objects' 2020-03-11T22:38:55.776 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:698: TEST_divergent_3: rados -p test put existing_2 /etc/resolv.conf 2020-03-11T22:38:55.776 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:702: TEST_divergent_3: seq 1 2 2020-03-11T22:38:55.776 INFO:tasks.workunit.client.0.smithi059.stdout:remove divergent objects 2020-03-11T22:38:55.776 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:702: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_REMOVE) 2020-03-11T22:38:55.777 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: expr 1 + 2 2020-03-11T22:38:55.777 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: rmi=3 2020-03-11T22:38:55.777 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:702: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_REMOVE) 2020-03-11T22:38:55.778 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: rados -p test rm existing_3 2020-03-11T22:38:55.778 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: expr 2 + 2 2020-03-11T22:38:55.779 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: rmi=4 2020-03-11T22:38:55.779 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:707: TEST_divergent_3: sleep 10 2020-03-11T22:38:55.779 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: rados -p test rm existing_4 2020-03-11T22:39:05.780 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:708: TEST_divergent_3: killall -9 rados 2020-03-11T22:39:05.784 INFO:tasks.workunit.client.0.smithi059.stderr:rados: no process found 2020-03-11T22:39:05.784 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:711: TEST_divergent_3: echo 'killing all the osds' 2020-03-11T22:39:05.785 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:712: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:39:05.785 INFO:tasks.workunit.client.0.smithi059.stdout:killing all the osds 2020-03-11T22:39:06.135 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:39:06.135 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 100 0 0 0 0 46628 0 0 16 16 active+clean 2020-03-11T22:38:31.478899+0000 31'106 31:118 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:39:06.135 INFO:tasks.workunit.client.0.smithi059.stdout: 2020-03-11T22:39:06.135 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:39:06.136 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2020-03-11T22:39:06.145 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:713: TEST_divergent_3: kill_daemons td/divergent-priors KILL osd 2020-03-11T22:39:06.145 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:39:06.146 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:39:06.146 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:39:06.146 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:39:06.146 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:39:06.251 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:39:06.252 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:714: TEST_divergent_3: for i in $osds 2020-03-11T22:39:06.252 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:716: TEST_divergent_3: ceph osd down osd.0 2020-03-11T22:39:06.744 INFO:tasks.workunit.client.0.smithi059.stderr:osd.0 is already down. 2020-03-11T22:39:06.752 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:714: TEST_divergent_3: for i in $osds 2020-03-11T22:39:06.752 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:716: TEST_divergent_3: ceph osd down osd.1 2020-03-11T22:39:07.319 INFO:tasks.workunit.client.0.smithi059.stderr:osd.1 is already down. 2020-03-11T22:39:07.326 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:714: TEST_divergent_3: for i in $osds 2020-03-11T22:39:07.327 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:716: TEST_divergent_3: ceph osd down osd.2 2020-03-11T22:39:07.859 INFO:tasks.workunit.client.0.smithi059.stderr:osd.2 is already down. 2020-03-11T22:39:07.867 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:718: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:39:07.868 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:720: TEST_divergent_3: ceph osd out osd.0 2020-03-11T22:39:08.380 INFO:tasks.workunit.client.0.smithi059.stderr:osd.0 is already out. 2020-03-11T22:39:08.402 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:718: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:39:08.402 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:720: TEST_divergent_3: ceph osd out osd.2 2020-03-11T22:39:08.937 INFO:tasks.workunit.client.0.smithi059.stderr:osd.2 is already out. 2020-03-11T22:39:08.945 INFO:tasks.workunit.client.0.smithi059.stdout:bringing up non_divergent 0 2 2020-03-11T22:39:08.945 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:724: TEST_divergent_3: echo 'bringing up non_divergent 0 2' 2020-03-11T22:39:08.946 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:725: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:39:09.287 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:39:09.287 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 100 0 0 0 0 46628 0 0 16 16 stale+active+clean 2020-03-11T22:38:31.478899+0000 31'106 31:118 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:39:09.288 INFO:tasks.workunit.client.0.smithi059.stdout: 2020-03-11T22:39:09.288 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:39:09.289 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2020-03-11T22:39:09.300 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:726: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:39:09.300 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:728: TEST_divergent_3: activate_osd td/divergent-priors 0 2020-03-11T22:39:09.300 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:39:09.301 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:39:09.301 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=0 2020-03-11T22:39:09.301 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:39:09.301 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/0 2020-03-11T22:39:09.301 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=cb7f2e42-1934-4145-a3c3-29bdd93f8adb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:39:09.301 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:39:09.301 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:39:09.302 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:39:09.302 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-11T22:39:09.302 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-11T22:39:09.302 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:39:09.302 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:39:09.302 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:39:09.302 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:39:09.303 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:39:09.303 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:39:09.303 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:39:09.303 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:39:09.303 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:39:09.303 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17931/$cluster-$name.asok' 2020-03-11T22:39:09.303 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' 2020-03-11T22:39:09.304 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:39:09.304 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:39:09.304 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:39:09.304 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:39:09.304 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:39:09.304 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:39:09.304 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:39:09.305 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:39:09.305 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/0 2020-03-11T22:39:09.305 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.0 2020-03-11T22:39:09.305 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 0 --fsid=cb7f2e42-1934-4145-a3c3-29bdd93f8adb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:39:09.305 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/0/whoami 2020-03-11T22:39:09.305 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.0 2020-03-11T22:39:09.306 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 0 = 0 ']' 2020-03-11T22:39:09.306 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:39:09.306 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:39:09.307 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:39:09.328 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:39:09.325+0000 7f97ae5b1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:09.329 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:39:09.327+0000 7f97ae5b1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:09.331 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:39:09.328+0000 7f97ae5b1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:09.666 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 0 2020-03-11T22:39:09.667 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:39:09.667 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:39:09.667 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:39:09.667 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:39:09.667 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:39:09.667 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:39:09.668 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:39:09.668 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:39:09.668 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:39:09.845 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:39:09.842+0000 7f97ae5b1ec0 -1 Falling back to public interface 2020-03-11T22:39:10.030 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:39:10.770 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:39:10.767+0000 7f97ae5b1ec0 -1 osd.0 31 log_to_monitors {default=true} 2020-03-11T22:39:11.031 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:39:11.032 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:39:11.032 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:39:11.032 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:39:11.032 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:39:11.032 INFO:tasks.workunit.client.0.smithi059.stdout:1 2020-03-11T22:39:11.411 INFO:tasks.workunit.client.0.smithi059.stdout:osd.0 up out weight 0 up_from 37 up_thru 0 down_at 32 last_clean_interval [5,31) [v2:127.0.0.1:6802/39814,v1:127.0.0.1:6803/39814] [v2:127.0.0.1:6804/39814,v1:127.0.0.1:6805/39814] exists,up 9e248648-07e4-4309-92d7-2283b32a88ea 2020-03-11T22:39:11.411 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.2 2020-03-11T22:39:11.411 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:39:11.411 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:39:11.411 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:39:11.411 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:726: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:39:11.412 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:728: TEST_divergent_3: activate_osd td/divergent-priors 2 2020-03-11T22:39:11.412 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:39:11.412 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:39:11.412 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=2 2020-03-11T22:39:11.412 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:39:11.412 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/2 2020-03-11T22:39:11.412 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=cb7f2e42-1934-4145-a3c3-29bdd93f8adb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:39:11.413 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:39:11.413 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:39:11.413 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:39:11.413 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-11T22:39:11.413 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-11T22:39:11.413 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:39:11.413 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:39:11.414 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:39:11.414 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:39:11.414 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:39:11.414 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:39:11.414 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:39:11.414 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:39:11.414 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:39:11.415 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17931/$cluster-$name.asok' 2020-03-11T22:39:11.415 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' 2020-03-11T22:39:11.415 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:39:11.415 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:39:11.415 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:39:11.415 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:39:11.415 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:39:11.416 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:39:11.416 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:39:11.416 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:39:11.417 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/2 2020-03-11T22:39:11.417 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.2 2020-03-11T22:39:11.417 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 2 --fsid=cb7f2e42-1934-4145-a3c3-29bdd93f8adb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:39:11.417 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/2/whoami 2020-03-11T22:39:11.418 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 2 = 2 ']' 2020-03-11T22:39:11.418 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:39:11.418 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:39:11.418 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:39:11.437 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:39:11.435+0000 7f47b42cdec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:11.439 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:39:11.437+0000 7f47b42cdec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:11.440 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:39:11.438+0000 7f47b42cdec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:11.795 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 2 2020-03-11T22:39:11.795 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:39:11.795 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:39:11.795 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:39:11.795 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:39:11.796 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:39:11.796 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:39:11.796 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:39:11.796 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:39:11.796 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:39:11.956 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:39:11.953+0000 7f47b42cdec0 -1 Falling back to public interface 2020-03-11T22:39:12.181 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:39:13.183 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:39:13.183 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:39:13.183 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:39:13.183 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:39:13.183 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:39:13.183 INFO:tasks.workunit.client.0.smithi059.stdout:1 2020-03-11T22:39:13.207 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:39:13.204+0000 7f47b42cdec0 -1 osd.2 31 log_to_monitors {default=true} 2020-03-11T22:39:13.565 INFO:tasks.workunit.client.0.smithi059.stdout:osd.2 up out weight 0 up_from 39 up_thru 0 down_at 34 last_clean_interval [15,31) [v2:127.0.0.1:6810/40008,v1:127.0.0.1:6811/40008] [v2:127.0.0.1:6812/40008,v1:127.0.0.1:6813/40008] exists,up 18b16e96-77ce-45af-9ce0-23852cb42c25 2020-03-11T22:39:13.565 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:39:13.565 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:39:13.565 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:39:13.565 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:730: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:39:13.566 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:732: TEST_divergent_3: ceph osd in osd.0 2020-03-11T22:39:14.138 INFO:tasks.workunit.client.0.smithi059.stderr:osd.0 is already in. 2020-03-11T22:39:14.146 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:730: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:39:14.147 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:732: TEST_divergent_3: ceph osd in osd.2 2020-03-11T22:39:14.670 INFO:tasks.workunit.client.0.smithi059.stderr:osd.2 is already in. 2020-03-11T22:39:14.679 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:735: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:39:14.679 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:735: TEST_divergent_3: wait_for_clean 2020-03-11T22:39:14.679 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:39:14.679 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:39:14.679 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:39:14.679 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:39:14.680 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:39:14.680 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:39:14.681 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:39:14.681 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:39:14.681 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:39:14.681 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:39:14.739 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:39:14.739 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:39:14.739 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:39:14.739 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:39:14.739 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:39:15.135 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:39:15.135 INFO:tasks.workunit.client.0.smithi059.stderr:1 2020-03-11T22:39:15.135 INFO:tasks.workunit.client.0.smithi059.stderr:2' 2020-03-11T22:39:15.136 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:39:15.136 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:39:15.136 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:39:15.274 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=158913789955 2020-03-11T22:39:15.274 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-158913789955' 2020-03-11T22:39:15.274 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:39:15.274 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:39:15.386 INFO:tasks.workunit.client.0.smithi059.stderr:Error ENXIO: problem getting command descriptions from osd.1 2020-03-11T22:39:15.394 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq= 2020-03-11T22:39:15.394 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-158913789955 1-' 2020-03-11T22:39:15.394 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:39:15.395 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:39:15.534 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=167503724547 2020-03-11T22:39:15.534 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-158913789955 1- 2-167503724547' 2020-03-11T22:39:15.535 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:39:15.535 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-158913789955 2020-03-11T22:39:15.535 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:39:15.536 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:39:15.536 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-158913789955 2020-03-11T22:39:15.536 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:39:15.537 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=158913789955 2020-03-11T22:39:15.537 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 158913789955' 2020-03-11T22:39:15.537 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.0 seq 158913789955 2020-03-11T22:39:15.538 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:39:15.902 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 0 -lt 158913789955 2020-03-11T22:39:15.902 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:39:16.903 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:39:16.903 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:39:17.282 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 158913789956 -lt 158913789955 2020-03-11T22:39:17.282 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:39:17.283 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1- 2020-03-11T22:39:17.283 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:39:17.283 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:39:17.284 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1- 2020-03-11T22:39:17.284 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:39:17.285 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq= 2020-03-11T22:39:17.285 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq ' 2020-03-11T22:39:17.285 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.1 seq 2020-03-11T22:39:17.286 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:39:17.674 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672975 -lt 2020-03-11T22:39:17.675 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 2079: test: 42949672975: unary operator expected 2020-03-11T22:39:17.675 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:39:17.675 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-167503724547 2020-03-11T22:39:17.675 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:39:17.676 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:39:17.676 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-167503724547 2020-03-11T22:39:17.676 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:39:17.677 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=167503724547 2020-03-11T22:39:17.678 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 167503724547' 2020-03-11T22:39:17.678 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:39:17.678 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.2 seq 167503724547 2020-03-11T22:39:18.042 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 167503724547 -lt 167503724547 2020-03-11T22:39:18.042 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:39:18.042 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:39:18.043 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:39:18.550 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 32 == 0 2020-03-11T22:39:18.551 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:39:18.551 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:39:18.551 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:39:18.551 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:39:18.551 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:39:18.551 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:39:18.552 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:39:18.908 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:39:18.908 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:39:18.909 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:39:18.909 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:39:19.392 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:39:19.393 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:39:19.393 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:39:19.393 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:39:19.393 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:39:19.393 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:39:19.493 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:39:19.494 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:39:19.495 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:39:19.495 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:39:19.495 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:39:19.495 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:39:19.495 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:39:19.495 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:39:19.871 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:39:19.871 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:39:19.871 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:39:19.872 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:39:20.344 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:39:20.344 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:39:20.344 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:39:20.344 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:39:20.345 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:39:20.345 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:39:20.345 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:39:20.345 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:39:20.346 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:39:20.817 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:39:20.818 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:39:20.818 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 1 >= 8 )) 2020-03-11T22:39:20.818 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:39:20.818 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.2 2020-03-11T22:39:21.018 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:39:21.020 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:39:21.021 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:39:21.021 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:39:21.022 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:39:21.022 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:39:21.022 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:39:21.022 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:39:21.378 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:39:21.378 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:39:21.378 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:39:21.379 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:39:21.857 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:39:21.858 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:39:21.858 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:39:21.858 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:39:21.858 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:39:21.858 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:39:21.858 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:39:21.858 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:39:21.859 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:39:22.334 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:39:22.335 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:39:22.335 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 2 >= 8 )) 2020-03-11T22:39:22.335 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:39:22.335 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.4 2020-03-11T22:39:22.736 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:39:22.736 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:39:22.737 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:39:22.737 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:39:22.737 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:39:22.737 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:39:22.737 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:39:22.737 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:39:23.098 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:39:23.099 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:39:23.099 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:39:23.099 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:39:23.569 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:39:23.570 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:39:23.570 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:39:23.570 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:39:23.570 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:39:23.570 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:39:23.570 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:39:23.571 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:39:23.571 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:39:24.047 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:39:24.047 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:39:24.047 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 3 >= 8 )) 2020-03-11T22:39:24.047 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:39:24.047 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.8 2020-03-11T22:39:24.847 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:39:24.848 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:39:24.849 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:39:24.849 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:39:24.849 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:39:24.849 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:39:24.849 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:39:24.849 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:39:25.239 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:39:25.239 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:39:25.239 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:39:25.240 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:39:25.730 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:39:25.730 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:39:25.730 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:39:25.730 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:39:25.730 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:39:25.731 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:39:25.731 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:39:25.731 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:39:25.731 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:39:26.195 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:39:26.196 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:39:26.196 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 4 >= 8 )) 2020-03-11T22:39:26.196 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:39:26.196 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 1.6 2020-03-11T22:39:27.800 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:39:27.800 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:39:27.801 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:39:27.801 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:39:27.801 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:39:27.801 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:39:27.802 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:39:27.802 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:39:28.176 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:39:28.177 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:39:28.177 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:39:28.177 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:39:28.637 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:39:28.637 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:39:28.638 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:39:28.638 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:39:28.638 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:39:28.638 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:39:28.638 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:39:28.639 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:39:28.639 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:39:29.132 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:39:29.133 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:39:29.133 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 5 >= 8 )) 2020-03-11T22:39:29.133 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:39:29.133 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 3.2 2020-03-11T22:39:32.333 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:39:32.334 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:39:32.335 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:39:32.335 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:39:32.335 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:39:32.336 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:39:32.336 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:39:32.336 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:39:32.702 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:39:32.703 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:39:32.703 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:39:32.703 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:39:33.180 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:39:33.180 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:39:33.180 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:39:33.180 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:39:33.180 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:39:33.181 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:39:33.181 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:39:33.181 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:39:33.181 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:39:33.669 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:39:33.669 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:39:33.669 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 6 >= 8 )) 2020-03-11T22:39:33.669 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:39:33.669 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 6.4 2020-03-11T22:39:40.070 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:39:40.070 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:39:40.071 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:39:40.071 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:39:40.071 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:39:40.071 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:39:40.071 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:39:40.072 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:39:40.440 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:39:40.440 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:39:40.440 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:39:40.441 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:39:40.917 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:39:40.917 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:39:40.918 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:39:40.918 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:39:40.918 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:39:40.918 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:39:40.918 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:39:40.918 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:39:40.918 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:39:41.392 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:39:41.392 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:39:41.392 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 7 >= 8 )) 2020-03-11T22:39:41.392 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:39:41.392 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 7.3 2020-03-11T22:39:48.698 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:39:48.698 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:39:48.699 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:39:48.699 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:39:48.699 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:39:48.699 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:39:48.699 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:39:48.699 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:39:49.065 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:39:49.066 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:39:49.066 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:39:49.066 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:39:49.541 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:39:49.541 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:39:49.541 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:39:49.542 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:39:49.542 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:39:49.542 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:39:49.542 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:39:49.543 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:39:49.543 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:39:50.012 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:39:50.012 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:39:50.013 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 8 >= 8 )) 2020-03-11T22:39:50.013 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: ceph report 2020-03-11T22:39:50.478 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2020-03-11T22:39:50.479 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_fingerprint": "4d1ca1d1-97bf-4599-82c8-e8c345704e49", 2020-03-11T22:39:50.479 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "15.1.0-1941-g4acc112", 2020-03-11T22:39:50.479 INFO:tasks.workunit.client.0.smithi059.stdout: "commit": "4acc1121e47039a3cdf2978ae458c10d153aa141", 2020-03-11T22:39:50.480 INFO:tasks.workunit.client.0.smithi059.stdout: "timestamp": "2020-03-11T22:39:50.475136+0000", 2020-03-11T22:39:50.480 INFO:tasks.workunit.client.0.smithi059.stdout: "tag": "", 2020-03-11T22:39:50.480 INFO:tasks.workunit.client.0.smithi059.stdout: "health": { 2020-03-11T22:39:50.480 INFO:tasks.workunit.client.0.smithi059.stdout: "status": "HEALTH_WARN", 2020-03-11T22:39:50.481 INFO:tasks.workunit.client.0.smithi059.stdout: "checks": { 2020-03-11T22:39:50.481 INFO:tasks.workunit.client.0.smithi059.stdout: "OSDMAP_FLAGS": { 2020-03-11T22:39:50.481 INFO:tasks.workunit.client.0.smithi059.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:39:50.481 INFO:tasks.workunit.client.0.smithi059.stdout: "summary": { 2020-03-11T22:39:50.482 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "nodown,noout,noin flag(s) set", 2020-03-11T22:39:50.482 INFO:tasks.workunit.client.0.smithi059.stdout: "count": 17 2020-03-11T22:39:50.482 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.482 INFO:tasks.workunit.client.0.smithi059.stdout: "detail": [], 2020-03-11T22:39:50.482 INFO:tasks.workunit.client.0.smithi059.stdout: "muted": false 2020-03-11T22:39:50.483 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.483 INFO:tasks.workunit.client.0.smithi059.stdout: "OSD_DOWN": { 2020-03-11T22:39:50.483 INFO:tasks.workunit.client.0.smithi059.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:39:50.483 INFO:tasks.workunit.client.0.smithi059.stdout: "summary": { 2020-03-11T22:39:50.484 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "1 osds down", 2020-03-11T22:39:50.484 INFO:tasks.workunit.client.0.smithi059.stdout: "count": 1 2020-03-11T22:39:50.484 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.484 INFO:tasks.workunit.client.0.smithi059.stdout: "detail": [ 2020-03-11T22:39:50.485 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.485 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "osd.1 (root=default,host=smithi059) is down" 2020-03-11T22:39:50.485 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:39:50.485 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:39:50.486 INFO:tasks.workunit.client.0.smithi059.stdout: "muted": false 2020-03-11T22:39:50.486 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.486 INFO:tasks.workunit.client.0.smithi059.stdout: "PG_DEGRADED": { 2020-03-11T22:39:50.486 INFO:tasks.workunit.client.0.smithi059.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:39:50.487 INFO:tasks.workunit.client.0.smithi059.stdout: "summary": { 2020-03-11T22:39:50.487 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 32 pgs degraded", 2020-03-11T22:39:50.487 INFO:tasks.workunit.client.0.smithi059.stdout: "count": 32 2020-03-11T22:39:50.487 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.488 INFO:tasks.workunit.client.0.smithi059.stdout: "detail": [ 2020-03-11T22:39:50.488 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.488 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:39:50.488 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.489 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.489 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "pg 1.1 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:39:50.489 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.489 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.490 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "pg 1.2 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:39:50.490 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.490 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.490 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "pg 1.3 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:39:50.491 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.491 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.491 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "pg 1.4 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:39:50.492 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.492 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.492 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "pg 1.5 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:39:50.492 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.493 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.493 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "pg 1.6 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:39:50.493 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.493 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.494 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "pg 1.7 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:39:50.494 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.494 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.494 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "pg 1.8 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:39:50.495 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.495 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.495 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "pg 1.9 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:39:50.495 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.495 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.496 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "pg 1.a is active+undersized+degraded, acting [0,2]" 2020-03-11T22:39:50.496 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.496 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.496 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "pg 1.b is active+undersized+degraded, acting [0,2]" 2020-03-11T22:39:50.497 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.497 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.497 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "pg 1.c is active+undersized+degraded, acting [0,2]" 2020-03-11T22:39:50.497 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.498 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.498 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "pg 1.d is active+undersized+degraded, acting [0,2]" 2020-03-11T22:39:50.498 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.498 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.499 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "pg 1.e is active+undersized+degraded, acting [0,2]" 2020-03-11T22:39:50.499 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.499 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.500 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "pg 1.f is active+undersized+degraded, acting [0,2]" 2020-03-11T22:39:50.500 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.500 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.500 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "pg 1.10 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:39:50.500 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.501 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.501 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "pg 1.11 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:39:50.501 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.501 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.502 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "pg 1.12 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:39:50.502 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.504 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.505 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "pg 1.13 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:39:50.505 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.505 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.505 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "pg 1.14 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:39:50.506 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.506 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.506 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "pg 1.15 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:39:50.506 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.507 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.507 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "pg 1.16 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:39:50.507 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.507 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.508 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "pg 1.17 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:39:50.508 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.508 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.508 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "pg 1.18 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:39:50.509 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.509 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.509 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "pg 1.19 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:39:50.509 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.510 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.510 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "pg 1.1a is active+undersized+degraded, acting [0,2]" 2020-03-11T22:39:50.510 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.510 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.511 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "pg 1.1b is active+undersized+degraded, acting [0,2]" 2020-03-11T22:39:50.511 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.511 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.511 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "pg 1.1c is active+undersized+degraded, acting [0,2]" 2020-03-11T22:39:50.511 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.512 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.512 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "pg 1.1d is active+undersized+degraded, acting [0,2]" 2020-03-11T22:39:50.512 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.512 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.513 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "pg 1.1e is active+undersized+degraded, acting [0,2]" 2020-03-11T22:39:50.513 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.513 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.513 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "pg 1.1f is active+undersized+degraded, acting [0,2]" 2020-03-11T22:39:50.514 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:39:50.514 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:39:50.515 INFO:tasks.workunit.client.0.smithi059.stdout: "muted": false 2020-03-11T22:39:50.515 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.515 INFO:tasks.workunit.client.0.smithi059.stdout: "POOL_APP_NOT_ENABLED": { 2020-03-11T22:39:50.515 INFO:tasks.workunit.client.0.smithi059.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:39:50.516 INFO:tasks.workunit.client.0.smithi059.stdout: "summary": { 2020-03-11T22:39:50.516 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "1 pool(s) do not have an application enabled", 2020-03-11T22:39:50.516 INFO:tasks.workunit.client.0.smithi059.stdout: "count": 1 2020-03-11T22:39:50.516 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.517 INFO:tasks.workunit.client.0.smithi059.stdout: "detail": [ 2020-03-11T22:39:50.517 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.517 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "application not enabled on pool 'test'" 2020-03-11T22:39:50.517 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.518 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.518 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2020-03-11T22:39:50.518 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:39:50.518 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:39:50.519 INFO:tasks.workunit.client.0.smithi059.stdout: "muted": false 2020-03-11T22:39:50.519 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:39:50.519 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.519 INFO:tasks.workunit.client.0.smithi059.stdout: "mutes": [] 2020-03-11T22:39:50.520 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.520 INFO:tasks.workunit.client.0.smithi059.stdout: "monmap_first_committed": 1, 2020-03-11T22:39:50.520 INFO:tasks.workunit.client.0.smithi059.stdout: "monmap_last_committed": 1, 2020-03-11T22:39:50.520 INFO:tasks.workunit.client.0.smithi059.stdout: "monmap": { 2020-03-11T22:39:50.521 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": 1, 2020-03-11T22:39:50.521 INFO:tasks.workunit.client.0.smithi059.stdout: "fsid": "cb7f2e42-1934-4145-a3c3-29bdd93f8adb", 2020-03-11T22:39:50.521 INFO:tasks.workunit.client.0.smithi059.stdout: "modified": "2020-03-11T22:38:07.922938Z", 2020-03-11T22:39:50.521 INFO:tasks.workunit.client.0.smithi059.stdout: "created": "2020-03-11T22:38:07.922938Z", 2020-03-11T22:39:50.522 INFO:tasks.workunit.client.0.smithi059.stdout: "min_mon_release": 15, 2020-03-11T22:39:50.522 INFO:tasks.workunit.client.0.smithi059.stdout: "min_mon_release_name": "octopus", 2020-03-11T22:39:50.522 INFO:tasks.workunit.client.0.smithi059.stdout: "features": { 2020-03-11T22:39:50.522 INFO:tasks.workunit.client.0.smithi059.stdout: "persistent": [ 2020-03-11T22:39:50.523 INFO:tasks.workunit.client.0.smithi059.stdout: "kraken", 2020-03-11T22:39:50.523 INFO:tasks.workunit.client.0.smithi059.stdout: "luminous", 2020-03-11T22:39:50.523 INFO:tasks.workunit.client.0.smithi059.stdout: "mimic", 2020-03-11T22:39:50.523 INFO:tasks.workunit.client.0.smithi059.stdout: "osdmap-prune", 2020-03-11T22:39:50.523 INFO:tasks.workunit.client.0.smithi059.stdout: "nautilus", 2020-03-11T22:39:50.524 INFO:tasks.workunit.client.0.smithi059.stdout: "octopus" 2020-03-11T22:39:50.524 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:39:50.524 INFO:tasks.workunit.client.0.smithi059.stdout: "optional": [] 2020-03-11T22:39:50.524 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.525 INFO:tasks.workunit.client.0.smithi059.stdout: "mons": [ 2020-03-11T22:39:50.525 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.525 INFO:tasks.workunit.client.0.smithi059.stdout: "rank": 0, 2020-03-11T22:39:50.525 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "a", 2020-03-11T22:39:50.526 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addrs": { 2020-03-11T22:39:50.526 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:39:50.526 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.526 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:39:50.526 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:7115", 2020-03-11T22:39:50.527 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 0 2020-03-11T22:39:50.527 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:39:50.527 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:39:50.527 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.528 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:7115/0", 2020-03-11T22:39:50.528 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addr": "127.0.0.1:7115/0", 2020-03-11T22:39:50.528 INFO:tasks.workunit.client.0.smithi059.stdout: "priority": 0, 2020-03-11T22:39:50.528 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 0 2020-03-11T22:39:50.529 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:39:50.529 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:39:50.529 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.529 INFO:tasks.workunit.client.0.smithi059.stdout: "quorum": [ 2020-03-11T22:39:50.530 INFO:tasks.workunit.client.0.smithi059.stdout: 0 2020-03-11T22:39:50.530 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:39:50.530 INFO:tasks.workunit.client.0.smithi059.stdout: "osdmap": { 2020-03-11T22:39:50.530 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": 44, 2020-03-11T22:39:50.530 INFO:tasks.workunit.client.0.smithi059.stdout: "fsid": "cb7f2e42-1934-4145-a3c3-29bdd93f8adb", 2020-03-11T22:39:50.531 INFO:tasks.workunit.client.0.smithi059.stdout: "created": "2020-03-11T22:38:08.591539+0000", 2020-03-11T22:39:50.531 INFO:tasks.workunit.client.0.smithi059.stdout: "modified": "2020-03-11T22:39:15.166894+0000", 2020-03-11T22:39:50.531 INFO:tasks.workunit.client.0.smithi059.stdout: "last_up_change": "2020-03-11T22:39:13.304952+0000", 2020-03-11T22:39:50.531 INFO:tasks.workunit.client.0.smithi059.stdout: "last_in_change": "2020-03-11T22:39:14.457686+0000", 2020-03-11T22:39:50.532 INFO:tasks.workunit.client.0.smithi059.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-03-11T22:39:50.532 INFO:tasks.workunit.client.0.smithi059.stdout: "flags_num": 5800384, 2020-03-11T22:39:50.532 INFO:tasks.workunit.client.0.smithi059.stdout: "flags_set": [ 2020-03-11T22:39:50.532 INFO:tasks.workunit.client.0.smithi059.stdout: "nodown", 2020-03-11T22:39:50.533 INFO:tasks.workunit.client.0.smithi059.stdout: "noin", 2020-03-11T22:39:50.533 INFO:tasks.workunit.client.0.smithi059.stdout: "noout", 2020-03-11T22:39:50.533 INFO:tasks.workunit.client.0.smithi059.stdout: "pglog_hardlimit", 2020-03-11T22:39:50.533 INFO:tasks.workunit.client.0.smithi059.stdout: "purged_snapdirs", 2020-03-11T22:39:50.533 INFO:tasks.workunit.client.0.smithi059.stdout: "recovery_deletes", 2020-03-11T22:39:50.534 INFO:tasks.workunit.client.0.smithi059.stdout: "sortbitwise" 2020-03-11T22:39:50.534 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:39:50.534 INFO:tasks.workunit.client.0.smithi059.stdout: "crush_version": 7, 2020-03-11T22:39:50.534 INFO:tasks.workunit.client.0.smithi059.stdout: "full_ratio": 0.99000000953674316, 2020-03-11T22:39:50.535 INFO:tasks.workunit.client.0.smithi059.stdout: "backfillfull_ratio": 0.99000000953674316, 2020-03-11T22:39:50.535 INFO:tasks.workunit.client.0.smithi059.stdout: "nearfull_ratio": 0.85000002384185791, 2020-03-11T22:39:50.535 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_snapshot": "", 2020-03-11T22:39:50.535 INFO:tasks.workunit.client.0.smithi059.stdout: "pool_max": 1, 2020-03-11T22:39:50.536 INFO:tasks.workunit.client.0.smithi059.stdout: "max_osd": 3, 2020-03-11T22:39:50.536 INFO:tasks.workunit.client.0.smithi059.stdout: "require_min_compat_client": "jewel", 2020-03-11T22:39:50.536 INFO:tasks.workunit.client.0.smithi059.stdout: "min_compat_client": "jewel", 2020-03-11T22:39:50.536 INFO:tasks.workunit.client.0.smithi059.stdout: "require_osd_release": "octopus", 2020-03-11T22:39:50.537 INFO:tasks.workunit.client.0.smithi059.stdout: "pools": [ 2020-03-11T22:39:50.537 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.537 INFO:tasks.workunit.client.0.smithi059.stdout: "pool": 1, 2020-03-11T22:39:50.537 INFO:tasks.workunit.client.0.smithi059.stdout: "pool_name": "test", 2020-03-11T22:39:50.538 INFO:tasks.workunit.client.0.smithi059.stdout: "create_time": "2020-03-11T22:38:30.749350+0000", 2020-03-11T22:39:50.538 INFO:tasks.workunit.client.0.smithi059.stdout: "flags": 1, 2020-03-11T22:39:50.538 INFO:tasks.workunit.client.0.smithi059.stdout: "flags_names": "hashpspool", 2020-03-11T22:39:50.538 INFO:tasks.workunit.client.0.smithi059.stdout: "type": 1, 2020-03-11T22:39:50.539 INFO:tasks.workunit.client.0.smithi059.stdout: "size": 3, 2020-03-11T22:39:50.539 INFO:tasks.workunit.client.0.smithi059.stdout: "min_size": 2, 2020-03-11T22:39:50.539 INFO:tasks.workunit.client.0.smithi059.stdout: "crush_rule": 0, 2020-03-11T22:39:50.539 INFO:tasks.workunit.client.0.smithi059.stdout: "object_hash": 2, 2020-03-11T22:39:50.540 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_autoscale_mode": "on", 2020-03-11T22:39:50.540 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_num": 32, 2020-03-11T22:39:50.540 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_placement_num": 1, 2020-03-11T22:39:50.540 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_placement_num_target": 32, 2020-03-11T22:39:50.541 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_num_target": 32, 2020-03-11T22:39:50.541 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_num_pending": 32, 2020-03-11T22:39:50.541 INFO:tasks.workunit.client.0.smithi059.stdout: "last_pg_merge_meta": { 2020-03-11T22:39:50.541 INFO:tasks.workunit.client.0.smithi059.stdout: "source_pgid": "0.0", 2020-03-11T22:39:50.542 INFO:tasks.workunit.client.0.smithi059.stdout: "ready_epoch": 0, 2020-03-11T22:39:50.542 INFO:tasks.workunit.client.0.smithi059.stdout: "last_epoch_started": 0, 2020-03-11T22:39:50.542 INFO:tasks.workunit.client.0.smithi059.stdout: "last_epoch_clean": 0, 2020-03-11T22:39:50.542 INFO:tasks.workunit.client.0.smithi059.stdout: "source_version": "0'0", 2020-03-11T22:39:50.542 INFO:tasks.workunit.client.0.smithi059.stdout: "target_version": "0'0" 2020-03-11T22:39:50.543 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.543 INFO:tasks.workunit.client.0.smithi059.stdout: "last_change": "43", 2020-03-11T22:39:50.543 INFO:tasks.workunit.client.0.smithi059.stdout: "last_force_op_resend": "0", 2020-03-11T22:39:50.543 INFO:tasks.workunit.client.0.smithi059.stdout: "last_force_op_resend_prenautilus": "0", 2020-03-11T22:39:50.544 INFO:tasks.workunit.client.0.smithi059.stdout: "last_force_op_resend_preluminous": "43", 2020-03-11T22:39:50.544 INFO:tasks.workunit.client.0.smithi059.stdout: "auid": 0, 2020-03-11T22:39:50.544 INFO:tasks.workunit.client.0.smithi059.stdout: "snap_mode": "selfmanaged", 2020-03-11T22:39:50.544 INFO:tasks.workunit.client.0.smithi059.stdout: "snap_seq": 0, 2020-03-11T22:39:50.545 INFO:tasks.workunit.client.0.smithi059.stdout: "snap_epoch": 0, 2020-03-11T22:39:50.545 INFO:tasks.workunit.client.0.smithi059.stdout: "pool_snaps": [], 2020-03-11T22:39:50.545 INFO:tasks.workunit.client.0.smithi059.stdout: "removed_snaps": "[]", 2020-03-11T22:39:50.545 INFO:tasks.workunit.client.0.smithi059.stdout: "quota_max_bytes": 0, 2020-03-11T22:39:50.546 INFO:tasks.workunit.client.0.smithi059.stdout: "quota_max_objects": 0, 2020-03-11T22:39:50.546 INFO:tasks.workunit.client.0.smithi059.stdout: "tiers": [], 2020-03-11T22:39:50.546 INFO:tasks.workunit.client.0.smithi059.stdout: "tier_of": -1, 2020-03-11T22:39:50.546 INFO:tasks.workunit.client.0.smithi059.stdout: "read_tier": -1, 2020-03-11T22:39:50.547 INFO:tasks.workunit.client.0.smithi059.stdout: "write_tier": -1, 2020-03-11T22:39:50.547 INFO:tasks.workunit.client.0.smithi059.stdout: "cache_mode": "none", 2020-03-11T22:39:50.547 INFO:tasks.workunit.client.0.smithi059.stdout: "target_max_bytes": 0, 2020-03-11T22:39:50.547 INFO:tasks.workunit.client.0.smithi059.stdout: "target_max_objects": 0, 2020-03-11T22:39:50.547 INFO:tasks.workunit.client.0.smithi059.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-03-11T22:39:50.548 INFO:tasks.workunit.client.0.smithi059.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-03-11T22:39:50.548 INFO:tasks.workunit.client.0.smithi059.stdout: "cache_target_full_ratio_micro": 800000, 2020-03-11T22:39:50.548 INFO:tasks.workunit.client.0.smithi059.stdout: "cache_min_flush_age": 0, 2020-03-11T22:39:50.549 INFO:tasks.workunit.client.0.smithi059.stdout: "cache_min_evict_age": 0, 2020-03-11T22:39:50.549 INFO:tasks.workunit.client.0.smithi059.stdout: "erasure_code_profile": "", 2020-03-11T22:39:50.549 INFO:tasks.workunit.client.0.smithi059.stdout: "hit_set_params": { 2020-03-11T22:39:50.549 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "none" 2020-03-11T22:39:50.550 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.550 INFO:tasks.workunit.client.0.smithi059.stdout: "hit_set_period": 0, 2020-03-11T22:39:50.550 INFO:tasks.workunit.client.0.smithi059.stdout: "hit_set_count": 0, 2020-03-11T22:39:50.550 INFO:tasks.workunit.client.0.smithi059.stdout: "use_gmt_hitset": true, 2020-03-11T22:39:50.550 INFO:tasks.workunit.client.0.smithi059.stdout: "min_read_recency_for_promote": 0, 2020-03-11T22:39:50.550 INFO:tasks.workunit.client.0.smithi059.stdout: "min_write_recency_for_promote": 0, 2020-03-11T22:39:50.551 INFO:tasks.workunit.client.0.smithi059.stdout: "hit_set_grade_decay_rate": 0, 2020-03-11T22:39:50.551 INFO:tasks.workunit.client.0.smithi059.stdout: "hit_set_search_last_n": 0, 2020-03-11T22:39:50.551 INFO:tasks.workunit.client.0.smithi059.stdout: "grade_table": [], 2020-03-11T22:39:50.551 INFO:tasks.workunit.client.0.smithi059.stdout: "stripe_width": 0, 2020-03-11T22:39:50.551 INFO:tasks.workunit.client.0.smithi059.stdout: "expected_num_objects": 0, 2020-03-11T22:39:50.551 INFO:tasks.workunit.client.0.smithi059.stdout: "fast_read": false, 2020-03-11T22:39:50.551 INFO:tasks.workunit.client.0.smithi059.stdout: "options": {}, 2020-03-11T22:39:50.552 INFO:tasks.workunit.client.0.smithi059.stdout: "application_metadata": {} 2020-03-11T22:39:50.552 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:39:50.552 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:39:50.552 INFO:tasks.workunit.client.0.smithi059.stdout: "osds": [ 2020-03-11T22:39:50.552 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.552 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 0, 2020-03-11T22:39:50.552 INFO:tasks.workunit.client.0.smithi059.stdout: "uuid": "9e248648-07e4-4309-92d7-2283b32a88ea", 2020-03-11T22:39:50.553 INFO:tasks.workunit.client.0.smithi059.stdout: "up": 1, 2020-03-11T22:39:50.553 INFO:tasks.workunit.client.0.smithi059.stdout: "in": 1, 2020-03-11T22:39:50.553 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 1, 2020-03-11T22:39:50.553 INFO:tasks.workunit.client.0.smithi059.stdout: "primary_affinity": 1, 2020-03-11T22:39:50.553 INFO:tasks.workunit.client.0.smithi059.stdout: "last_clean_begin": 5, 2020-03-11T22:39:50.553 INFO:tasks.workunit.client.0.smithi059.stdout: "last_clean_end": 31, 2020-03-11T22:39:50.553 INFO:tasks.workunit.client.0.smithi059.stdout: "up_from": 37, 2020-03-11T22:39:50.553 INFO:tasks.workunit.client.0.smithi059.stdout: "up_thru": 43, 2020-03-11T22:39:50.554 INFO:tasks.workunit.client.0.smithi059.stdout: "down_at": 32, 2020-03-11T22:39:50.554 INFO:tasks.workunit.client.0.smithi059.stdout: "lost_at": 0, 2020-03-11T22:39:50.554 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addrs": { 2020-03-11T22:39:50.554 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:39:50.554 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.554 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:39:50.554 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6802", 2020-03-11T22:39:50.555 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 39814 2020-03-11T22:39:50.555 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.555 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.555 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:39:50.555 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6803", 2020-03-11T22:39:50.555 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 39814 2020-03-11T22:39:50.555 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:39:50.556 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:39:50.556 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.556 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_addrs": { 2020-03-11T22:39:50.556 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:39:50.556 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.556 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:39:50.556 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6804", 2020-03-11T22:39:50.556 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 39814 2020-03-11T22:39:50.557 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.557 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.557 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:39:50.557 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6805", 2020-03-11T22:39:50.557 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 39814 2020-03-11T22:39:50.557 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:39:50.557 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:39:50.558 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.558 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_back_addrs": { 2020-03-11T22:39:50.558 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:39:50.558 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.558 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:39:50.558 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6808", 2020-03-11T22:39:50.558 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 39814 2020-03-11T22:39:50.559 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.559 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.559 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:39:50.559 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6809", 2020-03-11T22:39:50.559 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 39814 2020-03-11T22:39:50.559 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:39:50.559 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:39:50.560 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.560 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_front_addrs": { 2020-03-11T22:39:50.560 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:39:50.560 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.560 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:39:50.560 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6806", 2020-03-11T22:39:50.560 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 39814 2020-03-11T22:39:50.560 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.561 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.561 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:39:50.561 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6807", 2020-03-11T22:39:50.561 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 39814 2020-03-11T22:39:50.561 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:39:50.561 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:39:50.561 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.562 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addr": "127.0.0.1:6803/39814", 2020-03-11T22:39:50.562 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_addr": "127.0.0.1:6805/39814", 2020-03-11T22:39:50.562 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_back_addr": "127.0.0.1:6809/39814", 2020-03-11T22:39:50.562 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_front_addr": "127.0.0.1:6807/39814", 2020-03-11T22:39:50.562 INFO:tasks.workunit.client.0.smithi059.stdout: "state": [ 2020-03-11T22:39:50.562 INFO:tasks.workunit.client.0.smithi059.stdout: "exists", 2020-03-11T22:39:50.562 INFO:tasks.workunit.client.0.smithi059.stdout: "up" 2020-03-11T22:39:50.562 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:39:50.563 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.563 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.563 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 1, 2020-03-11T22:39:50.563 INFO:tasks.workunit.client.0.smithi059.stdout: "uuid": "98cb8180-5961-424c-a790-9139892a46ae", 2020-03-11T22:39:50.563 INFO:tasks.workunit.client.0.smithi059.stdout: "up": 0, 2020-03-11T22:39:50.563 INFO:tasks.workunit.client.0.smithi059.stdout: "in": 1, 2020-03-11T22:39:50.563 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 1, 2020-03-11T22:39:50.564 INFO:tasks.workunit.client.0.smithi059.stdout: "primary_affinity": 1, 2020-03-11T22:39:50.564 INFO:tasks.workunit.client.0.smithi059.stdout: "last_clean_begin": 0, 2020-03-11T22:39:50.564 INFO:tasks.workunit.client.0.smithi059.stdout: "last_clean_end": 0, 2020-03-11T22:39:50.564 INFO:tasks.workunit.client.0.smithi059.stdout: "up_from": 10, 2020-03-11T22:39:50.564 INFO:tasks.workunit.client.0.smithi059.stdout: "up_thru": 23, 2020-03-11T22:39:50.564 INFO:tasks.workunit.client.0.smithi059.stdout: "down_at": 33, 2020-03-11T22:39:50.564 INFO:tasks.workunit.client.0.smithi059.stdout: "lost_at": 0, 2020-03-11T22:39:50.565 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addrs": { 2020-03-11T22:39:50.565 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:39:50.565 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.565 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:39:50.565 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:39:50.565 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 35398 2020-03-11T22:39:50.565 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.565 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.566 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:39:50.566 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:39:50.566 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 35398 2020-03-11T22:39:50.566 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:39:50.566 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:39:50.566 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.566 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_addrs": { 2020-03-11T22:39:50.567 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:39:50.567 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.567 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:39:50.567 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:39:50.567 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 35398 2020-03-11T22:39:50.567 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.567 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.567 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:39:50.568 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:39:50.568 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 35398 2020-03-11T22:39:50.568 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:39:50.568 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:39:50.568 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.568 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_back_addrs": { 2020-03-11T22:39:50.568 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:39:50.569 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.569 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:39:50.569 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:39:50.569 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 35398 2020-03-11T22:39:50.569 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.569 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.569 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:39:50.569 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:39:50.570 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 35398 2020-03-11T22:39:50.570 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:39:50.570 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:39:50.570 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.570 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_front_addrs": { 2020-03-11T22:39:50.570 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:39:50.570 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.571 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:39:50.571 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:39:50.571 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 35398 2020-03-11T22:39:50.571 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.571 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.571 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:39:50.571 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:39:50.571 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 35398 2020-03-11T22:39:50.572 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:39:50.572 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:39:50.572 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.572 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addr": "127.0.0.1:6811/35398", 2020-03-11T22:39:50.572 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_addr": "127.0.0.1:6813/35398", 2020-03-11T22:39:50.572 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_back_addr": "127.0.0.1:6817/35398", 2020-03-11T22:39:50.572 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_front_addr": "127.0.0.1:6815/35398", 2020-03-11T22:39:50.573 INFO:tasks.workunit.client.0.smithi059.stdout: "state": [ 2020-03-11T22:39:50.573 INFO:tasks.workunit.client.0.smithi059.stdout: "exists" 2020-03-11T22:39:50.573 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:39:50.573 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.573 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.573 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 2, 2020-03-11T22:39:50.573 INFO:tasks.workunit.client.0.smithi059.stdout: "uuid": "18b16e96-77ce-45af-9ce0-23852cb42c25", 2020-03-11T22:39:50.574 INFO:tasks.workunit.client.0.smithi059.stdout: "up": 1, 2020-03-11T22:39:50.574 INFO:tasks.workunit.client.0.smithi059.stdout: "in": 1, 2020-03-11T22:39:50.574 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 1, 2020-03-11T22:39:50.574 INFO:tasks.workunit.client.0.smithi059.stdout: "primary_affinity": 1, 2020-03-11T22:39:50.574 INFO:tasks.workunit.client.0.smithi059.stdout: "last_clean_begin": 15, 2020-03-11T22:39:50.574 INFO:tasks.workunit.client.0.smithi059.stdout: "last_clean_end": 31, 2020-03-11T22:39:50.574 INFO:tasks.workunit.client.0.smithi059.stdout: "up_from": 39, 2020-03-11T22:39:50.575 INFO:tasks.workunit.client.0.smithi059.stdout: "up_thru": 0, 2020-03-11T22:39:50.575 INFO:tasks.workunit.client.0.smithi059.stdout: "down_at": 34, 2020-03-11T22:39:50.575 INFO:tasks.workunit.client.0.smithi059.stdout: "lost_at": 0, 2020-03-11T22:39:50.575 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addrs": { 2020-03-11T22:39:50.575 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:39:50.575 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.575 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:39:50.575 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:39:50.576 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 40008 2020-03-11T22:39:50.576 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.576 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.576 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:39:50.576 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:39:50.576 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 40008 2020-03-11T22:39:50.576 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:39:50.577 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:39:50.577 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.577 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_addrs": { 2020-03-11T22:39:50.577 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:39:50.577 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.577 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:39:50.577 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:39:50.578 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 40008 2020-03-11T22:39:50.578 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.578 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.578 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:39:50.578 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:39:50.578 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 40008 2020-03-11T22:39:50.578 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:39:50.579 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:39:50.579 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.579 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_back_addrs": { 2020-03-11T22:39:50.579 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:39:50.579 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.579 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:39:50.579 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:39:50.579 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 40008 2020-03-11T22:39:50.580 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.580 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.580 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:39:50.580 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:39:50.580 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 40008 2020-03-11T22:39:50.580 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:39:50.580 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:39:50.581 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.581 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_front_addrs": { 2020-03-11T22:39:50.581 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:39:50.581 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.581 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:39:50.581 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:39:50.581 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 40008 2020-03-11T22:39:50.582 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.582 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.582 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:39:50.582 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:39:50.582 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 40008 2020-03-11T22:39:50.582 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:39:50.582 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:39:50.583 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.583 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addr": "127.0.0.1:6811/40008", 2020-03-11T22:39:50.583 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_addr": "127.0.0.1:6813/40008", 2020-03-11T22:39:50.583 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_back_addr": "127.0.0.1:6817/40008", 2020-03-11T22:39:50.583 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_front_addr": "127.0.0.1:6815/40008", 2020-03-11T22:39:50.583 INFO:tasks.workunit.client.0.smithi059.stdout: "state": [ 2020-03-11T22:39:50.583 INFO:tasks.workunit.client.0.smithi059.stdout: "exists", 2020-03-11T22:39:50.584 INFO:tasks.workunit.client.0.smithi059.stdout: "up" 2020-03-11T22:39:50.584 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:39:50.584 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:39:50.584 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:39:50.584 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_xinfo": [ 2020-03-11T22:39:50.584 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.584 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 0, 2020-03-11T22:39:50.584 INFO:tasks.workunit.client.0.smithi059.stdout: "down_stamp": "2020-03-11T22:39:06.563045+0000", 2020-03-11T22:39:50.585 INFO:tasks.workunit.client.0.smithi059.stdout: "laggy_probability": 0, 2020-03-11T22:39:50.585 INFO:tasks.workunit.client.0.smithi059.stdout: "laggy_interval": 0, 2020-03-11T22:39:50.585 INFO:tasks.workunit.client.0.smithi059.stdout: "features": 4540138292836696063, 2020-03-11T22:39:50.585 INFO:tasks.workunit.client.0.smithi059.stdout: "old_weight": 0, 2020-03-11T22:39:50.585 INFO:tasks.workunit.client.0.smithi059.stdout: "last_purged_snaps_scrub": "2020-03-11T22:38:15.567036+0000", 2020-03-11T22:39:50.585 INFO:tasks.workunit.client.0.smithi059.stdout: "dead_epoch": 0 2020-03-11T22:39:50.585 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.586 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.586 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 1, 2020-03-11T22:39:50.586 INFO:tasks.workunit.client.0.smithi059.stdout: "down_stamp": "2020-03-11T22:39:07.122111+0000", 2020-03-11T22:39:50.586 INFO:tasks.workunit.client.0.smithi059.stdout: "laggy_probability": 0, 2020-03-11T22:39:50.586 INFO:tasks.workunit.client.0.smithi059.stdout: "laggy_interval": 0, 2020-03-11T22:39:50.586 INFO:tasks.workunit.client.0.smithi059.stdout: "features": 4540138292836696063, 2020-03-11T22:39:50.586 INFO:tasks.workunit.client.0.smithi059.stdout: "old_weight": 0, 2020-03-11T22:39:50.586 INFO:tasks.workunit.client.0.smithi059.stdout: "last_purged_snaps_scrub": "2020-03-11T22:38:21.900248+0000", 2020-03-11T22:39:50.587 INFO:tasks.workunit.client.0.smithi059.stdout: "dead_epoch": 0 2020-03-11T22:39:50.587 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.587 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.587 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 2, 2020-03-11T22:39:50.587 INFO:tasks.workunit.client.0.smithi059.stdout: "down_stamp": "2020-03-11T22:39:07.637108+0000", 2020-03-11T22:39:50.587 INFO:tasks.workunit.client.0.smithi059.stdout: "laggy_probability": 0, 2020-03-11T22:39:50.587 INFO:tasks.workunit.client.0.smithi059.stdout: "laggy_interval": 0, 2020-03-11T22:39:50.588 INFO:tasks.workunit.client.0.smithi059.stdout: "features": 4540138292836696063, 2020-03-11T22:39:50.588 INFO:tasks.workunit.client.0.smithi059.stdout: "old_weight": 0, 2020-03-11T22:39:50.588 INFO:tasks.workunit.client.0.smithi059.stdout: "last_purged_snaps_scrub": "2020-03-11T22:38:28.351224+0000", 2020-03-11T22:39:50.588 INFO:tasks.workunit.client.0.smithi059.stdout: "dead_epoch": 0 2020-03-11T22:39:50.588 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:39:50.588 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:39:50.588 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_upmap": [], 2020-03-11T22:39:50.588 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_upmap_items": [], 2020-03-11T22:39:50.589 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_temp": [], 2020-03-11T22:39:50.589 INFO:tasks.workunit.client.0.smithi059.stdout: "primary_temp": [], 2020-03-11T22:39:50.589 INFO:tasks.workunit.client.0.smithi059.stdout: "blacklist": {}, 2020-03-11T22:39:50.589 INFO:tasks.workunit.client.0.smithi059.stdout: "erasure_code_profiles": { 2020-03-11T22:39:50.589 INFO:tasks.workunit.client.0.smithi059.stdout: "default": { 2020-03-11T22:39:50.589 INFO:tasks.workunit.client.0.smithi059.stdout: "k": "2", 2020-03-11T22:39:50.589 INFO:tasks.workunit.client.0.smithi059.stdout: "m": "2", 2020-03-11T22:39:50.590 INFO:tasks.workunit.client.0.smithi059.stdout: "plugin": "jerasure", 2020-03-11T22:39:50.590 INFO:tasks.workunit.client.0.smithi059.stdout: "technique": "reed_sol_van" 2020-03-11T22:39:50.590 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:39:50.590 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.590 INFO:tasks.workunit.client.0.smithi059.stdout: "removed_snaps_queue": [], 2020-03-11T22:39:50.590 INFO:tasks.workunit.client.0.smithi059.stdout: "new_removed_snaps": [], 2020-03-11T22:39:50.590 INFO:tasks.workunit.client.0.smithi059.stdout: "new_purged_snaps": [], 2020-03-11T22:39:50.591 INFO:tasks.workunit.client.0.smithi059.stdout: "crush_node_flags": {}, 2020-03-11T22:39:50.591 INFO:tasks.workunit.client.0.smithi059.stdout: "device_class_flags": {} 2020-03-11T22:39:50.591 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.591 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_metadata": [ 2020-03-11T22:39:50.591 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.591 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 0, 2020-03-11T22:39:50.591 INFO:tasks.workunit.client.0.smithi059.stdout: "arch": "x86_64", 2020-03-11T22:39:50.592 INFO:tasks.workunit.client.0.smithi059.stdout: "back_addr": "[v2:127.0.0.1:6804/39814,v1:127.0.0.1:6805/39814]", 2020-03-11T22:39:50.592 INFO:tasks.workunit.client.0.smithi059.stdout: "back_iface": "", 2020-03-11T22:39:50.592 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs": "1", 2020-03-11T22:39:50.592 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:39:50.592 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:39:50.593 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:39:50.593 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:39:50.593 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:39:50.593 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:39:50.593 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:39:50.593 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2020-03-11T22:39:50.593 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:39:50.594 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:39:50.594 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:39:50.594 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:39:50.594 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_release": "octopus", 2020-03-11T22:39:50.594 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:39:50.594 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:39:50.594 INFO:tasks.workunit.client.0.smithi059.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-03-11T22:39:50.594 INFO:tasks.workunit.client.0.smithi059.stdout: "default_device_class": "hdd", 2020-03-11T22:39:50.595 INFO:tasks.workunit.client.0.smithi059.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4J2P7", 2020-03-11T22:39:50.595 INFO:tasks.workunit.client.0.smithi059.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:39:50.595 INFO:tasks.workunit.client.0.smithi059.stdout: "devices": "sda", 2020-03-11T22:39:50.595 INFO:tasks.workunit.client.0.smithi059.stdout: "distro": "centos", 2020-03-11T22:39:50.595 INFO:tasks.workunit.client.0.smithi059.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:39:50.595 INFO:tasks.workunit.client.0.smithi059.stdout: "distro_version": "8", 2020-03-11T22:39:50.595 INFO:tasks.workunit.client.0.smithi059.stdout: "front_addr": "[v2:127.0.0.1:6802/39814,v1:127.0.0.1:6803/39814]", 2020-03-11T22:39:50.596 INFO:tasks.workunit.client.0.smithi059.stdout: "front_iface": "", 2020-03-11T22:39:50.596 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/39814,v1:127.0.0.1:6809/39814]", 2020-03-11T22:39:50.596 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/39814,v1:127.0.0.1:6807/39814]", 2020-03-11T22:39:50.596 INFO:tasks.workunit.client.0.smithi059.stdout: "hostname": "smithi059", 2020-03-11T22:39:50.596 INFO:tasks.workunit.client.0.smithi059.stdout: "journal_rotational": "1", 2020-03-11T22:39:50.596 INFO:tasks.workunit.client.0.smithi059.stdout: "kernel_description": "#1 SMP Wed Feb 5 02:00:39 UTC 2020", 2020-03-11T22:39:50.596 INFO:tasks.workunit.client.0.smithi059.stdout: "kernel_version": "4.18.0-147.5.1.el8_1.x86_64", 2020-03-11T22:39:50.597 INFO:tasks.workunit.client.0.smithi059.stdout: "mem_swap_kb": "0", 2020-03-11T22:39:50.597 INFO:tasks.workunit.client.0.smithi059.stdout: "mem_total_kb": "32662084", 2020-03-11T22:39:50.597 INFO:tasks.workunit.client.0.smithi059.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:39:50.597 INFO:tasks.workunit.client.0.smithi059.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:39:50.597 INFO:tasks.workunit.client.0.smithi059.stdout: "os": "Linux", 2020-03-11T22:39:50.597 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_data": "td/divergent-priors/0", 2020-03-11T22:39:50.597 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:39:50.598 INFO:tasks.workunit.client.0.smithi059.stdout: "rotational": "1" 2020-03-11T22:39:50.598 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.598 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.598 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 1, 2020-03-11T22:39:50.598 INFO:tasks.workunit.client.0.smithi059.stdout: "arch": "x86_64", 2020-03-11T22:39:50.598 INFO:tasks.workunit.client.0.smithi059.stdout: "back_addr": "[v2:127.0.0.1:6812/35398,v1:127.0.0.1:6813/35398]", 2020-03-11T22:39:50.598 INFO:tasks.workunit.client.0.smithi059.stdout: "back_iface": "", 2020-03-11T22:39:50.599 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs": "1", 2020-03-11T22:39:50.599 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:39:50.599 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:39:50.599 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:39:50.599 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:39:50.599 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:39:50.599 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:39:50.600 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:39:50.600 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2020-03-11T22:39:50.600 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:39:50.600 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:39:50.600 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:39:50.600 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:39:50.601 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_release": "octopus", 2020-03-11T22:39:50.601 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:39:50.601 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:39:50.601 INFO:tasks.workunit.client.0.smithi059.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-03-11T22:39:50.601 INFO:tasks.workunit.client.0.smithi059.stdout: "default_device_class": "hdd", 2020-03-11T22:39:50.601 INFO:tasks.workunit.client.0.smithi059.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4J2P7", 2020-03-11T22:39:50.601 INFO:tasks.workunit.client.0.smithi059.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:39:50.602 INFO:tasks.workunit.client.0.smithi059.stdout: "devices": "sda", 2020-03-11T22:39:50.602 INFO:tasks.workunit.client.0.smithi059.stdout: "distro": "centos", 2020-03-11T22:39:50.602 INFO:tasks.workunit.client.0.smithi059.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:39:50.602 INFO:tasks.workunit.client.0.smithi059.stdout: "distro_version": "8", 2020-03-11T22:39:50.602 INFO:tasks.workunit.client.0.smithi059.stdout: "front_addr": "[v2:127.0.0.1:6810/35398,v1:127.0.0.1:6811/35398]", 2020-03-11T22:39:50.602 INFO:tasks.workunit.client.0.smithi059.stdout: "front_iface": "", 2020-03-11T22:39:50.602 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/35398,v1:127.0.0.1:6817/35398]", 2020-03-11T22:39:50.603 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/35398,v1:127.0.0.1:6815/35398]", 2020-03-11T22:39:50.603 INFO:tasks.workunit.client.0.smithi059.stdout: "hostname": "smithi059", 2020-03-11T22:39:50.603 INFO:tasks.workunit.client.0.smithi059.stdout: "journal_rotational": "1", 2020-03-11T22:39:50.603 INFO:tasks.workunit.client.0.smithi059.stdout: "kernel_description": "#1 SMP Wed Feb 5 02:00:39 UTC 2020", 2020-03-11T22:39:50.603 INFO:tasks.workunit.client.0.smithi059.stdout: "kernel_version": "4.18.0-147.5.1.el8_1.x86_64", 2020-03-11T22:39:50.603 INFO:tasks.workunit.client.0.smithi059.stdout: "mem_swap_kb": "0", 2020-03-11T22:39:50.603 INFO:tasks.workunit.client.0.smithi059.stdout: "mem_total_kb": "32662084", 2020-03-11T22:39:50.604 INFO:tasks.workunit.client.0.smithi059.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:39:50.604 INFO:tasks.workunit.client.0.smithi059.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:39:50.604 INFO:tasks.workunit.client.0.smithi059.stdout: "os": "Linux", 2020-03-11T22:39:50.604 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_data": "td/divergent-priors/1", 2020-03-11T22:39:50.604 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:39:50.604 INFO:tasks.workunit.client.0.smithi059.stdout: "rotational": "1" 2020-03-11T22:39:50.604 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.604 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.605 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 2, 2020-03-11T22:39:50.605 INFO:tasks.workunit.client.0.smithi059.stdout: "arch": "x86_64", 2020-03-11T22:39:50.605 INFO:tasks.workunit.client.0.smithi059.stdout: "back_addr": "[v2:127.0.0.1:6812/40008,v1:127.0.0.1:6813/40008]", 2020-03-11T22:39:50.605 INFO:tasks.workunit.client.0.smithi059.stdout: "back_iface": "", 2020-03-11T22:39:50.605 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs": "1", 2020-03-11T22:39:50.605 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:39:50.605 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:39:50.606 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:39:50.606 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:39:50.606 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:39:50.606 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:39:50.606 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:39:50.606 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2020-03-11T22:39:50.606 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:39:50.607 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:39:50.607 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:39:50.607 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:39:50.607 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_release": "octopus", 2020-03-11T22:39:50.607 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:39:50.607 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:39:50.607 INFO:tasks.workunit.client.0.smithi059.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-03-11T22:39:50.608 INFO:tasks.workunit.client.0.smithi059.stdout: "default_device_class": "hdd", 2020-03-11T22:39:50.608 INFO:tasks.workunit.client.0.smithi059.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4J2P7", 2020-03-11T22:39:50.608 INFO:tasks.workunit.client.0.smithi059.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:39:50.608 INFO:tasks.workunit.client.0.smithi059.stdout: "devices": "sda", 2020-03-11T22:39:50.608 INFO:tasks.workunit.client.0.smithi059.stdout: "distro": "centos", 2020-03-11T22:39:50.608 INFO:tasks.workunit.client.0.smithi059.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:39:50.608 INFO:tasks.workunit.client.0.smithi059.stdout: "distro_version": "8", 2020-03-11T22:39:50.609 INFO:tasks.workunit.client.0.smithi059.stdout: "front_addr": "[v2:127.0.0.1:6810/40008,v1:127.0.0.1:6811/40008]", 2020-03-11T22:39:50.609 INFO:tasks.workunit.client.0.smithi059.stdout: "front_iface": "", 2020-03-11T22:39:50.609 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/40008,v1:127.0.0.1:6817/40008]", 2020-03-11T22:39:50.609 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/40008,v1:127.0.0.1:6815/40008]", 2020-03-11T22:39:50.609 INFO:tasks.workunit.client.0.smithi059.stdout: "hostname": "smithi059", 2020-03-11T22:39:50.609 INFO:tasks.workunit.client.0.smithi059.stdout: "journal_rotational": "1", 2020-03-11T22:39:50.609 INFO:tasks.workunit.client.0.smithi059.stdout: "kernel_description": "#1 SMP Wed Feb 5 02:00:39 UTC 2020", 2020-03-11T22:39:50.609 INFO:tasks.workunit.client.0.smithi059.stdout: "kernel_version": "4.18.0-147.5.1.el8_1.x86_64", 2020-03-11T22:39:50.610 INFO:tasks.workunit.client.0.smithi059.stdout: "mem_swap_kb": "0", 2020-03-11T22:39:50.610 INFO:tasks.workunit.client.0.smithi059.stdout: "mem_total_kb": "32662084", 2020-03-11T22:39:50.610 INFO:tasks.workunit.client.0.smithi059.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:39:50.610 INFO:tasks.workunit.client.0.smithi059.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:39:50.610 INFO:tasks.workunit.client.0.smithi059.stdout: "os": "Linux", 2020-03-11T22:39:50.610 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_data": "td/divergent-priors/2", 2020-03-11T22:39:50.610 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:39:50.611 INFO:tasks.workunit.client.0.smithi059.stdout: "rotational": "1" 2020-03-11T22:39:50.611 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:39:50.611 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:39:50.611 INFO:tasks.workunit.client.0.smithi059.stdout: "osdmap_first_committed": 1, 2020-03-11T22:39:50.611 INFO:tasks.workunit.client.0.smithi059.stdout: "osdmap_last_committed": 44, 2020-03-11T22:39:50.611 INFO:tasks.workunit.client.0.smithi059.stdout: "crushmap": { 2020-03-11T22:39:50.611 INFO:tasks.workunit.client.0.smithi059.stdout: "devices": [ 2020-03-11T22:39:50.612 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.612 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 0, 2020-03-11T22:39:50.612 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "osd.0", 2020-03-11T22:39:50.612 INFO:tasks.workunit.client.0.smithi059.stdout: "class": "hdd" 2020-03-11T22:39:50.612 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.612 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.612 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 1, 2020-03-11T22:39:50.613 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "osd.1", 2020-03-11T22:39:50.613 INFO:tasks.workunit.client.0.smithi059.stdout: "class": "hdd" 2020-03-11T22:39:50.613 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.613 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.613 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 2, 2020-03-11T22:39:50.613 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "osd.2", 2020-03-11T22:39:50.613 INFO:tasks.workunit.client.0.smithi059.stdout: "class": "hdd" 2020-03-11T22:39:50.613 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:39:50.614 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:39:50.614 INFO:tasks.workunit.client.0.smithi059.stdout: "types": [ 2020-03-11T22:39:50.614 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.614 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 0, 2020-03-11T22:39:50.614 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "osd" 2020-03-11T22:39:50.614 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.614 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.615 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 1, 2020-03-11T22:39:50.615 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "host" 2020-03-11T22:39:50.615 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.615 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.615 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 2, 2020-03-11T22:39:50.616 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "chassis" 2020-03-11T22:39:50.616 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.616 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.616 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 3, 2020-03-11T22:39:50.616 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "rack" 2020-03-11T22:39:50.616 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.616 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.617 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 4, 2020-03-11T22:39:50.617 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "row" 2020-03-11T22:39:50.617 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.617 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.617 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 5, 2020-03-11T22:39:50.617 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "pdu" 2020-03-11T22:39:50.617 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.618 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.618 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 6, 2020-03-11T22:39:50.618 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "pod" 2020-03-11T22:39:50.618 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.618 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.618 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 7, 2020-03-11T22:39:50.618 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "room" 2020-03-11T22:39:50.618 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.619 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.619 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 8, 2020-03-11T22:39:50.619 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "datacenter" 2020-03-11T22:39:50.619 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.619 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.619 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 9, 2020-03-11T22:39:50.619 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "zone" 2020-03-11T22:39:50.620 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.620 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.620 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 10, 2020-03-11T22:39:50.620 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "region" 2020-03-11T22:39:50.620 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.620 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.620 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 11, 2020-03-11T22:39:50.620 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "root" 2020-03-11T22:39:50.621 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:39:50.621 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:39:50.621 INFO:tasks.workunit.client.0.smithi059.stdout: "buckets": [ 2020-03-11T22:39:50.621 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.621 INFO:tasks.workunit.client.0.smithi059.stdout: "id": -1, 2020-03-11T22:39:50.621 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "default", 2020-03-11T22:39:50.621 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 11, 2020-03-11T22:39:50.622 INFO:tasks.workunit.client.0.smithi059.stdout: "type_name": "root", 2020-03-11T22:39:50.622 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 19206, 2020-03-11T22:39:50.622 INFO:tasks.workunit.client.0.smithi059.stdout: "alg": "straw2", 2020-03-11T22:39:50.622 INFO:tasks.workunit.client.0.smithi059.stdout: "hash": "rjenkins1", 2020-03-11T22:39:50.622 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2020-03-11T22:39:50.622 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.622 INFO:tasks.workunit.client.0.smithi059.stdout: "id": -3, 2020-03-11T22:39:50.623 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 19206, 2020-03-11T22:39:50.623 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 0 2020-03-11T22:39:50.623 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:39:50.623 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:39:50.623 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.623 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.623 INFO:tasks.workunit.client.0.smithi059.stdout: "id": -2, 2020-03-11T22:39:50.623 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "default~hdd", 2020-03-11T22:39:50.624 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 11, 2020-03-11T22:39:50.624 INFO:tasks.workunit.client.0.smithi059.stdout: "type_name": "root", 2020-03-11T22:39:50.624 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 19206, 2020-03-11T22:39:50.624 INFO:tasks.workunit.client.0.smithi059.stdout: "alg": "straw2", 2020-03-11T22:39:50.624 INFO:tasks.workunit.client.0.smithi059.stdout: "hash": "rjenkins1", 2020-03-11T22:39:50.624 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2020-03-11T22:39:50.624 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.625 INFO:tasks.workunit.client.0.smithi059.stdout: "id": -4, 2020-03-11T22:39:50.625 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 19206, 2020-03-11T22:39:50.625 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 0 2020-03-11T22:39:50.625 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:39:50.625 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:39:50.625 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.625 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.625 INFO:tasks.workunit.client.0.smithi059.stdout: "id": -3, 2020-03-11T22:39:50.626 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "smithi059", 2020-03-11T22:39:50.626 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 1, 2020-03-11T22:39:50.626 INFO:tasks.workunit.client.0.smithi059.stdout: "type_name": "host", 2020-03-11T22:39:50.626 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 19206, 2020-03-11T22:39:50.626 INFO:tasks.workunit.client.0.smithi059.stdout: "alg": "straw2", 2020-03-11T22:39:50.626 INFO:tasks.workunit.client.0.smithi059.stdout: "hash": "rjenkins1", 2020-03-11T22:39:50.626 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2020-03-11T22:39:50.627 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.627 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 0, 2020-03-11T22:39:50.627 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 6402, 2020-03-11T22:39:50.627 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 0 2020-03-11T22:39:50.627 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.627 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.627 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 1, 2020-03-11T22:39:50.628 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 6402, 2020-03-11T22:39:50.628 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 1 2020-03-11T22:39:50.628 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.628 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.628 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 2, 2020-03-11T22:39:50.628 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 6402, 2020-03-11T22:39:50.628 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 2 2020-03-11T22:39:50.628 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:39:50.629 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:39:50.629 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.629 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.629 INFO:tasks.workunit.client.0.smithi059.stdout: "id": -4, 2020-03-11T22:39:50.629 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "smithi059~hdd", 2020-03-11T22:39:50.629 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 1, 2020-03-11T22:39:50.629 INFO:tasks.workunit.client.0.smithi059.stdout: "type_name": "host", 2020-03-11T22:39:50.630 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 19206, 2020-03-11T22:39:50.630 INFO:tasks.workunit.client.0.smithi059.stdout: "alg": "straw2", 2020-03-11T22:39:50.630 INFO:tasks.workunit.client.0.smithi059.stdout: "hash": "rjenkins1", 2020-03-11T22:39:50.630 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2020-03-11T22:39:50.630 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.630 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 0, 2020-03-11T22:39:50.630 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 6402, 2020-03-11T22:39:50.630 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 0 2020-03-11T22:39:50.631 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.631 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.631 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 1, 2020-03-11T22:39:50.631 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 6402, 2020-03-11T22:39:50.631 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 1 2020-03-11T22:39:50.631 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.631 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.632 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 2, 2020-03-11T22:39:50.632 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 6402, 2020-03-11T22:39:50.632 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 2 2020-03-11T22:39:50.632 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:39:50.632 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:39:50.632 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:39:50.632 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:39:50.632 INFO:tasks.workunit.client.0.smithi059.stdout: "rules": [ 2020-03-11T22:39:50.633 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.633 INFO:tasks.workunit.client.0.smithi059.stdout: "rule_id": 0, 2020-03-11T22:39:50.633 INFO:tasks.workunit.client.0.smithi059.stdout: "rule_name": "replicated_rule", 2020-03-11T22:39:50.633 INFO:tasks.workunit.client.0.smithi059.stdout: "ruleset": 0, 2020-03-11T22:39:50.633 INFO:tasks.workunit.client.0.smithi059.stdout: "type": 1, 2020-03-11T22:39:50.633 INFO:tasks.workunit.client.0.smithi059.stdout: "min_size": 1, 2020-03-11T22:39:50.633 INFO:tasks.workunit.client.0.smithi059.stdout: "max_size": 10, 2020-03-11T22:39:50.634 INFO:tasks.workunit.client.0.smithi059.stdout: "steps": [ 2020-03-11T22:39:50.634 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.634 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "take", 2020-03-11T22:39:50.634 INFO:tasks.workunit.client.0.smithi059.stdout: "item": -1, 2020-03-11T22:39:50.634 INFO:tasks.workunit.client.0.smithi059.stdout: "item_name": "default" 2020-03-11T22:39:50.634 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.634 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.634 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "choose_firstn", 2020-03-11T22:39:50.635 INFO:tasks.workunit.client.0.smithi059.stdout: "num": 0, 2020-03-11T22:39:50.635 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "osd" 2020-03-11T22:39:50.635 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.635 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.635 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "emit" 2020-03-11T22:39:50.635 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:39:50.635 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:39:50.636 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:39:50.636 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:39:50.636 INFO:tasks.workunit.client.0.smithi059.stdout: "tunables": { 2020-03-11T22:39:50.636 INFO:tasks.workunit.client.0.smithi059.stdout: "choose_local_tries": 0, 2020-03-11T22:39:50.636 INFO:tasks.workunit.client.0.smithi059.stdout: "choose_local_fallback_tries": 0, 2020-03-11T22:39:50.636 INFO:tasks.workunit.client.0.smithi059.stdout: "choose_total_tries": 50, 2020-03-11T22:39:50.636 INFO:tasks.workunit.client.0.smithi059.stdout: "chooseleaf_descend_once": 1, 2020-03-11T22:39:50.637 INFO:tasks.workunit.client.0.smithi059.stdout: "chooseleaf_vary_r": 1, 2020-03-11T22:39:50.637 INFO:tasks.workunit.client.0.smithi059.stdout: "chooseleaf_stable": 1, 2020-03-11T22:39:50.637 INFO:tasks.workunit.client.0.smithi059.stdout: "straw_calc_version": 1, 2020-03-11T22:39:50.637 INFO:tasks.workunit.client.0.smithi059.stdout: "allowed_bucket_algs": 54, 2020-03-11T22:39:50.637 INFO:tasks.workunit.client.0.smithi059.stdout: "profile": "jewel", 2020-03-11T22:39:50.637 INFO:tasks.workunit.client.0.smithi059.stdout: "optimal_tunables": 1, 2020-03-11T22:39:50.637 INFO:tasks.workunit.client.0.smithi059.stdout: "legacy_tunables": 0, 2020-03-11T22:39:50.638 INFO:tasks.workunit.client.0.smithi059.stdout: "minimum_required_version": "jewel", 2020-03-11T22:39:50.638 INFO:tasks.workunit.client.0.smithi059.stdout: "require_feature_tunables": 1, 2020-03-11T22:39:50.638 INFO:tasks.workunit.client.0.smithi059.stdout: "require_feature_tunables2": 1, 2020-03-11T22:39:50.638 INFO:tasks.workunit.client.0.smithi059.stdout: "has_v2_rules": 0, 2020-03-11T22:39:50.638 INFO:tasks.workunit.client.0.smithi059.stdout: "require_feature_tunables3": 1, 2020-03-11T22:39:50.638 INFO:tasks.workunit.client.0.smithi059.stdout: "has_v3_rules": 0, 2020-03-11T22:39:50.638 INFO:tasks.workunit.client.0.smithi059.stdout: "has_v4_buckets": 1, 2020-03-11T22:39:50.639 INFO:tasks.workunit.client.0.smithi059.stdout: "require_feature_tunables5": 1, 2020-03-11T22:39:50.639 INFO:tasks.workunit.client.0.smithi059.stdout: "has_v5_rules": 0 2020-03-11T22:39:50.639 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.639 INFO:tasks.workunit.client.0.smithi059.stdout: "choose_args": {} 2020-03-11T22:39:50.639 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.639 INFO:tasks.workunit.client.0.smithi059.stdout: "fsmap": { 2020-03-11T22:39:50.639 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": 1, 2020-03-11T22:39:50.640 INFO:tasks.workunit.client.0.smithi059.stdout: "default_fscid": -1, 2020-03-11T22:39:50.640 INFO:tasks.workunit.client.0.smithi059.stdout: "compat": { 2020-03-11T22:39:50.640 INFO:tasks.workunit.client.0.smithi059.stdout: "compat": {}, 2020-03-11T22:39:50.640 INFO:tasks.workunit.client.0.smithi059.stdout: "ro_compat": {}, 2020-03-11T22:39:50.640 INFO:tasks.workunit.client.0.smithi059.stdout: "incompat": { 2020-03-11T22:39:50.640 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_1": "base v0.20", 2020-03-11T22:39:50.640 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_2": "client writeable ranges", 2020-03-11T22:39:50.641 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_3": "default file layouts on dirs", 2020-03-11T22:39:50.641 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_4": "dir inode in separate object", 2020-03-11T22:39:50.641 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_5": "mds uses versioned encoding", 2020-03-11T22:39:50.641 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_6": "dirfrag is stored in omap", 2020-03-11T22:39:50.641 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_8": "no anchor table", 2020-03-11T22:39:50.641 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_9": "file layout v2", 2020-03-11T22:39:50.641 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_10": "snaprealm v2" 2020-03-11T22:39:50.642 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:39:50.642 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.642 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_flags": { 2020-03-11T22:39:50.642 INFO:tasks.workunit.client.0.smithi059.stdout: "enable_multiple": false, 2020-03-11T22:39:50.642 INFO:tasks.workunit.client.0.smithi059.stdout: "ever_enabled_multiple": false 2020-03-11T22:39:50.642 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.642 INFO:tasks.workunit.client.0.smithi059.stdout: "standbys": [], 2020-03-11T22:39:50.642 INFO:tasks.workunit.client.0.smithi059.stdout: "filesystems": [] 2020-03-11T22:39:50.643 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.643 INFO:tasks.workunit.client.0.smithi059.stdout: "mdsmap_first_committed": 1, 2020-03-11T22:39:50.643 INFO:tasks.workunit.client.0.smithi059.stdout: "mdsmap_last_committed": 1, 2020-03-11T22:39:50.643 INFO:tasks.workunit.client.0.smithi059.stdout: "auth": { 2020-03-11T22:39:50.643 INFO:tasks.workunit.client.0.smithi059.stdout: "first_committed": 1, 2020-03-11T22:39:50.643 INFO:tasks.workunit.client.0.smithi059.stdout: "last_committed": 5, 2020-03-11T22:39:50.643 INFO:tasks.workunit.client.0.smithi059.stdout: "num_secrets": 3 2020-03-11T22:39:50.644 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.644 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pg": 32, 2020-03-11T22:39:50.644 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pg_active": 32, 2020-03-11T22:39:50.644 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pg_unknown": 0, 2020-03-11T22:39:50.644 INFO:tasks.workunit.client.0.smithi059.stdout: "num_osd": 3, 2020-03-11T22:39:50.644 INFO:tasks.workunit.client.0.smithi059.stdout: "pool_sum": { 2020-03-11T22:39:50.644 INFO:tasks.workunit.client.0.smithi059.stdout: "stat_sum": { 2020-03-11T22:39:50.644 INFO:tasks.workunit.client.0.smithi059.stdout: "num_bytes": 48300, 2020-03-11T22:39:50.645 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects": 100, 2020-03-11T22:39:50.645 INFO:tasks.workunit.client.0.smithi059.stdout: "num_object_clones": 0, 2020-03-11T22:39:50.645 INFO:tasks.workunit.client.0.smithi059.stdout: "num_object_copies": 300, 2020-03-11T22:39:50.645 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:39:50.645 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_missing": 0, 2020-03-11T22:39:50.645 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_degraded": 100, 2020-03-11T22:39:50.645 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_misplaced": 0, 2020-03-11T22:39:50.646 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_unfound": 0, 2020-03-11T22:39:50.646 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_dirty": 100, 2020-03-11T22:39:50.646 INFO:tasks.workunit.client.0.smithi059.stdout: "num_whiteouts": 0, 2020-03-11T22:39:50.646 INFO:tasks.workunit.client.0.smithi059.stdout: "num_read": 0, 2020-03-11T22:39:50.646 INFO:tasks.workunit.client.0.smithi059.stdout: "num_read_kb": 0, 2020-03-11T22:39:50.646 INFO:tasks.workunit.client.0.smithi059.stdout: "num_write": 100, 2020-03-11T22:39:50.646 INFO:tasks.workunit.client.0.smithi059.stdout: "num_write_kb": 100, 2020-03-11T22:39:50.647 INFO:tasks.workunit.client.0.smithi059.stdout: "num_scrub_errors": 0, 2020-03-11T22:39:50.647 INFO:tasks.workunit.client.0.smithi059.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:39:50.647 INFO:tasks.workunit.client.0.smithi059.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:39:50.647 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_recovered": 0, 2020-03-11T22:39:50.647 INFO:tasks.workunit.client.0.smithi059.stdout: "num_bytes_recovered": 0, 2020-03-11T22:39:50.647 INFO:tasks.workunit.client.0.smithi059.stdout: "num_keys_recovered": 0, 2020-03-11T22:39:50.647 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_omap": 0, 2020-03-11T22:39:50.647 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:39:50.648 INFO:tasks.workunit.client.0.smithi059.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:39:50.648 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush": 0, 2020-03-11T22:39:50.648 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush_kb": 0, 2020-03-11T22:39:50.648 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict": 0, 2020-03-11T22:39:50.648 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict_kb": 0, 2020-03-11T22:39:50.648 INFO:tasks.workunit.client.0.smithi059.stdout: "num_promote": 0, 2020-03-11T22:39:50.648 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush_mode_high": 0, 2020-03-11T22:39:50.649 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush_mode_low": 0, 2020-03-11T22:39:50.649 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict_mode_some": 0, 2020-03-11T22:39:50.649 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict_mode_full": 0, 2020-03-11T22:39:50.649 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_pinned": 0, 2020-03-11T22:39:50.649 INFO:tasks.workunit.client.0.smithi059.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:39:50.649 INFO:tasks.workunit.client.0.smithi059.stdout: "num_large_omap_objects": 0, 2020-03-11T22:39:50.649 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_manifest": 0, 2020-03-11T22:39:50.650 INFO:tasks.workunit.client.0.smithi059.stdout: "num_omap_bytes": 0, 2020-03-11T22:39:50.650 INFO:tasks.workunit.client.0.smithi059.stdout: "num_omap_keys": 0, 2020-03-11T22:39:50.650 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_repaired": 0 2020-03-11T22:39:50.650 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.650 INFO:tasks.workunit.client.0.smithi059.stdout: "store_stats": { 2020-03-11T22:39:50.650 INFO:tasks.workunit.client.0.smithi059.stdout: "total": 0, 2020-03-11T22:39:50.651 INFO:tasks.workunit.client.0.smithi059.stdout: "available": 0, 2020-03-11T22:39:50.651 INFO:tasks.workunit.client.0.smithi059.stdout: "internally_reserved": 0, 2020-03-11T22:39:50.651 INFO:tasks.workunit.client.0.smithi059.stdout: "allocated": 0, 2020-03-11T22:39:50.651 INFO:tasks.workunit.client.0.smithi059.stdout: "data_stored": 0, 2020-03-11T22:39:50.651 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed": 0, 2020-03-11T22:39:50.651 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_allocated": 0, 2020-03-11T22:39:50.651 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_original": 0, 2020-03-11T22:39:50.652 INFO:tasks.workunit.client.0.smithi059.stdout: "omap_allocated": 0, 2020-03-11T22:39:50.652 INFO:tasks.workunit.client.0.smithi059.stdout: "internal_metadata": 0 2020-03-11T22:39:50.652 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.652 INFO:tasks.workunit.client.0.smithi059.stdout: "log_size": 512, 2020-03-11T22:39:50.652 INFO:tasks.workunit.client.0.smithi059.stdout: "ondisk_log_size": 512, 2020-03-11T22:39:50.652 INFO:tasks.workunit.client.0.smithi059.stdout: "up": 64, 2020-03-11T22:39:50.652 INFO:tasks.workunit.client.0.smithi059.stdout: "acting": 64, 2020-03-11T22:39:50.653 INFO:tasks.workunit.client.0.smithi059.stdout: "num_store_stats": 0 2020-03-11T22:39:50.653 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.653 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_sum": { 2020-03-11T22:39:50.653 INFO:tasks.workunit.client.0.smithi059.stdout: "up_from": 0, 2020-03-11T22:39:50.653 INFO:tasks.workunit.client.0.smithi059.stdout: "seq": 0, 2020-03-11T22:39:50.653 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pgs": 65, 2020-03-11T22:39:50.653 INFO:tasks.workunit.client.0.smithi059.stdout: "num_osds": 3, 2020-03-11T22:39:50.653 INFO:tasks.workunit.client.0.smithi059.stdout: "num_per_pool_osds": 3, 2020-03-11T22:39:50.654 INFO:tasks.workunit.client.0.smithi059.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:39:50.654 INFO:tasks.workunit.client.0.smithi059.stdout: "kb": 314572800, 2020-03-11T22:39:50.654 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used": 3180672, 2020-03-11T22:39:50.654 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used_data": 34624, 2020-03-11T22:39:50.654 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used_omap": 7, 2020-03-11T22:39:50.654 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used_meta": 3145720, 2020-03-11T22:39:50.654 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_avail": 311392128, 2020-03-11T22:39:50.654 INFO:tasks.workunit.client.0.smithi059.stdout: "statfs": { 2020-03-11T22:39:50.655 INFO:tasks.workunit.client.0.smithi059.stdout: "total": 322122547200, 2020-03-11T22:39:50.655 INFO:tasks.workunit.client.0.smithi059.stdout: "available": 318865539072, 2020-03-11T22:39:50.655 INFO:tasks.workunit.client.0.smithi059.stdout: "internally_reserved": 3221225472, 2020-03-11T22:39:50.655 INFO:tasks.workunit.client.0.smithi059.stdout: "allocated": 35454976, 2020-03-11T22:39:50.655 INFO:tasks.workunit.client.0.smithi059.stdout: "data_stored": 438648, 2020-03-11T22:39:50.655 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed": 0, 2020-03-11T22:39:50.655 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_allocated": 0, 2020-03-11T22:39:50.656 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_original": 0, 2020-03-11T22:39:50.656 INFO:tasks.workunit.client.0.smithi059.stdout: "omap_allocated": 8190, 2020-03-11T22:39:50.656 INFO:tasks.workunit.client.0.smithi059.stdout: "internal_metadata": 3221217282 2020-03-11T22:39:50.656 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.656 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_peers": [], 2020-03-11T22:39:50.656 INFO:tasks.workunit.client.0.smithi059.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:39:50.656 INFO:tasks.workunit.client.0.smithi059.stdout: "num_snap_trimming": 0, 2020-03-11T22:39:50.657 INFO:tasks.workunit.client.0.smithi059.stdout: "num_shards_repaired": 0, 2020-03-11T22:39:50.657 INFO:tasks.workunit.client.0.smithi059.stdout: "op_queue_age_hist": { 2020-03-11T22:39:50.657 INFO:tasks.workunit.client.0.smithi059.stdout: "histogram": [], 2020-03-11T22:39:50.657 INFO:tasks.workunit.client.0.smithi059.stdout: "upper_bound": 1 2020-03-11T22:39:50.657 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.657 INFO:tasks.workunit.client.0.smithi059.stdout: "perf_stat": { 2020-03-11T22:39:50.657 INFO:tasks.workunit.client.0.smithi059.stdout: "commit_latency_ms": 0, 2020-03-11T22:39:50.658 INFO:tasks.workunit.client.0.smithi059.stdout: "apply_latency_ms": 0, 2020-03-11T22:39:50.658 INFO:tasks.workunit.client.0.smithi059.stdout: "commit_latency_ns": 0, 2020-03-11T22:39:50.658 INFO:tasks.workunit.client.0.smithi059.stdout: "apply_latency_ns": 0 2020-03-11T22:39:50.658 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.658 INFO:tasks.workunit.client.0.smithi059.stdout: "alerts": [], 2020-03-11T22:39:50.658 INFO:tasks.workunit.client.0.smithi059.stdout: "network_ping_times": [] 2020-03-11T22:39:50.658 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.658 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_sum_by_class": { 2020-03-11T22:39:50.659 INFO:tasks.workunit.client.0.smithi059.stdout: "hdd": { 2020-03-11T22:39:50.659 INFO:tasks.workunit.client.0.smithi059.stdout: "up_from": 0, 2020-03-11T22:39:50.659 INFO:tasks.workunit.client.0.smithi059.stdout: "seq": 0, 2020-03-11T22:39:50.659 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pgs": 65, 2020-03-11T22:39:50.659 INFO:tasks.workunit.client.0.smithi059.stdout: "num_osds": 3, 2020-03-11T22:39:50.659 INFO:tasks.workunit.client.0.smithi059.stdout: "num_per_pool_osds": 3, 2020-03-11T22:39:50.659 INFO:tasks.workunit.client.0.smithi059.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:39:50.660 INFO:tasks.workunit.client.0.smithi059.stdout: "kb": 314572800, 2020-03-11T22:39:50.660 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used": 3180672, 2020-03-11T22:39:50.660 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used_data": 34624, 2020-03-11T22:39:50.660 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used_omap": 7, 2020-03-11T22:39:50.660 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used_meta": 3145720, 2020-03-11T22:39:50.660 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_avail": 311392128, 2020-03-11T22:39:50.660 INFO:tasks.workunit.client.0.smithi059.stdout: "statfs": { 2020-03-11T22:39:50.661 INFO:tasks.workunit.client.0.smithi059.stdout: "total": 322122547200, 2020-03-11T22:39:50.661 INFO:tasks.workunit.client.0.smithi059.stdout: "available": 318865539072, 2020-03-11T22:39:50.661 INFO:tasks.workunit.client.0.smithi059.stdout: "internally_reserved": 3221225472, 2020-03-11T22:39:50.661 INFO:tasks.workunit.client.0.smithi059.stdout: "allocated": 35454976, 2020-03-11T22:39:50.661 INFO:tasks.workunit.client.0.smithi059.stdout: "data_stored": 438648, 2020-03-11T22:39:50.661 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed": 0, 2020-03-11T22:39:50.661 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_allocated": 0, 2020-03-11T22:39:50.661 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_original": 0, 2020-03-11T22:39:50.662 INFO:tasks.workunit.client.0.smithi059.stdout: "omap_allocated": 8190, 2020-03-11T22:39:50.662 INFO:tasks.workunit.client.0.smithi059.stdout: "internal_metadata": 3221217282 2020-03-11T22:39:50.662 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.662 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_peers": [], 2020-03-11T22:39:50.662 INFO:tasks.workunit.client.0.smithi059.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:39:50.662 INFO:tasks.workunit.client.0.smithi059.stdout: "num_snap_trimming": 0, 2020-03-11T22:39:50.662 INFO:tasks.workunit.client.0.smithi059.stdout: "num_shards_repaired": 0, 2020-03-11T22:39:50.663 INFO:tasks.workunit.client.0.smithi059.stdout: "op_queue_age_hist": { 2020-03-11T22:39:50.663 INFO:tasks.workunit.client.0.smithi059.stdout: "histogram": [], 2020-03-11T22:39:50.663 INFO:tasks.workunit.client.0.smithi059.stdout: "upper_bound": 1 2020-03-11T22:39:50.663 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.663 INFO:tasks.workunit.client.0.smithi059.stdout: "perf_stat": { 2020-03-11T22:39:50.663 INFO:tasks.workunit.client.0.smithi059.stdout: "commit_latency_ms": 0, 2020-03-11T22:39:50.663 INFO:tasks.workunit.client.0.smithi059.stdout: "apply_latency_ms": 0, 2020-03-11T22:39:50.664 INFO:tasks.workunit.client.0.smithi059.stdout: "commit_latency_ns": 0, 2020-03-11T22:39:50.664 INFO:tasks.workunit.client.0.smithi059.stdout: "apply_latency_ns": 0 2020-03-11T22:39:50.664 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.664 INFO:tasks.workunit.client.0.smithi059.stdout: "alerts": [], 2020-03-11T22:39:50.664 INFO:tasks.workunit.client.0.smithi059.stdout: "network_ping_times": [] 2020-03-11T22:39:50.664 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:39:50.664 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.664 INFO:tasks.workunit.client.0.smithi059.stdout: "pool_stats": [ 2020-03-11T22:39:50.665 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.665 INFO:tasks.workunit.client.0.smithi059.stdout: "poolid": 1, 2020-03-11T22:39:50.665 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pg": 32, 2020-03-11T22:39:50.665 INFO:tasks.workunit.client.0.smithi059.stdout: "stat_sum": { 2020-03-11T22:39:50.665 INFO:tasks.workunit.client.0.smithi059.stdout: "num_bytes": 48300, 2020-03-11T22:39:50.665 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects": 100, 2020-03-11T22:39:50.665 INFO:tasks.workunit.client.0.smithi059.stdout: "num_object_clones": 0, 2020-03-11T22:39:50.666 INFO:tasks.workunit.client.0.smithi059.stdout: "num_object_copies": 300, 2020-03-11T22:39:50.666 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:39:50.666 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_missing": 0, 2020-03-11T22:39:50.666 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_degraded": 100, 2020-03-11T22:39:50.666 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_misplaced": 0, 2020-03-11T22:39:50.666 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_unfound": 0, 2020-03-11T22:39:50.666 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_dirty": 100, 2020-03-11T22:39:50.666 INFO:tasks.workunit.client.0.smithi059.stdout: "num_whiteouts": 0, 2020-03-11T22:39:50.667 INFO:tasks.workunit.client.0.smithi059.stdout: "num_read": 0, 2020-03-11T22:39:50.667 INFO:tasks.workunit.client.0.smithi059.stdout: "num_read_kb": 0, 2020-03-11T22:39:50.667 INFO:tasks.workunit.client.0.smithi059.stdout: "num_write": 100, 2020-03-11T22:39:50.667 INFO:tasks.workunit.client.0.smithi059.stdout: "num_write_kb": 100, 2020-03-11T22:39:50.667 INFO:tasks.workunit.client.0.smithi059.stdout: "num_scrub_errors": 0, 2020-03-11T22:39:50.667 INFO:tasks.workunit.client.0.smithi059.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:39:50.667 INFO:tasks.workunit.client.0.smithi059.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:39:50.667 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_recovered": 0, 2020-03-11T22:39:50.668 INFO:tasks.workunit.client.0.smithi059.stdout: "num_bytes_recovered": 0, 2020-03-11T22:39:50.668 INFO:tasks.workunit.client.0.smithi059.stdout: "num_keys_recovered": 0, 2020-03-11T22:39:50.668 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_omap": 0, 2020-03-11T22:39:50.668 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:39:50.668 INFO:tasks.workunit.client.0.smithi059.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:39:50.668 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush": 0, 2020-03-11T22:39:50.668 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush_kb": 0, 2020-03-11T22:39:50.669 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict": 0, 2020-03-11T22:39:50.669 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict_kb": 0, 2020-03-11T22:39:50.669 INFO:tasks.workunit.client.0.smithi059.stdout: "num_promote": 0, 2020-03-11T22:39:50.669 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush_mode_high": 0, 2020-03-11T22:39:50.669 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush_mode_low": 0, 2020-03-11T22:39:50.669 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict_mode_some": 0, 2020-03-11T22:39:50.669 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict_mode_full": 0, 2020-03-11T22:39:50.670 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_pinned": 0, 2020-03-11T22:39:50.670 INFO:tasks.workunit.client.0.smithi059.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:39:50.670 INFO:tasks.workunit.client.0.smithi059.stdout: "num_large_omap_objects": 0, 2020-03-11T22:39:50.670 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_manifest": 0, 2020-03-11T22:39:50.670 INFO:tasks.workunit.client.0.smithi059.stdout: "num_omap_bytes": 0, 2020-03-11T22:39:50.670 INFO:tasks.workunit.client.0.smithi059.stdout: "num_omap_keys": 0, 2020-03-11T22:39:50.670 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_repaired": 0 2020-03-11T22:39:50.670 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.671 INFO:tasks.workunit.client.0.smithi059.stdout: "store_stats": { 2020-03-11T22:39:50.671 INFO:tasks.workunit.client.0.smithi059.stdout: "total": 0, 2020-03-11T22:39:50.671 INFO:tasks.workunit.client.0.smithi059.stdout: "available": 0, 2020-03-11T22:39:50.671 INFO:tasks.workunit.client.0.smithi059.stdout: "internally_reserved": 0, 2020-03-11T22:39:50.671 INFO:tasks.workunit.client.0.smithi059.stdout: "allocated": 19529728, 2020-03-11T22:39:50.671 INFO:tasks.workunit.client.0.smithi059.stdout: "data_stored": 143228, 2020-03-11T22:39:50.671 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed": 0, 2020-03-11T22:39:50.672 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_allocated": 0, 2020-03-11T22:39:50.672 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_original": 0, 2020-03-11T22:39:50.672 INFO:tasks.workunit.client.0.smithi059.stdout: "omap_allocated": 0, 2020-03-11T22:39:50.672 INFO:tasks.workunit.client.0.smithi059.stdout: "internal_metadata": 0 2020-03-11T22:39:50.672 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.672 INFO:tasks.workunit.client.0.smithi059.stdout: "log_size": 512, 2020-03-11T22:39:50.672 INFO:tasks.workunit.client.0.smithi059.stdout: "ondisk_log_size": 512, 2020-03-11T22:39:50.672 INFO:tasks.workunit.client.0.smithi059.stdout: "up": 64, 2020-03-11T22:39:50.673 INFO:tasks.workunit.client.0.smithi059.stdout: "acting": 64, 2020-03-11T22:39:50.673 INFO:tasks.workunit.client.0.smithi059.stdout: "num_store_stats": 3 2020-03-11T22:39:50.673 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:39:50.673 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:39:50.673 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_stats": [ 2020-03-11T22:39:50.673 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.673 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 0, 2020-03-11T22:39:50.674 INFO:tasks.workunit.client.0.smithi059.stdout: "seq": 158913789962 2020-03-11T22:39:50.674 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.674 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.674 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 1, 2020-03-11T22:39:50.674 INFO:tasks.workunit.client.0.smithi059.stdout: "seq": 42949672975 2020-03-11T22:39:50.674 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.674 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.674 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 2, 2020-03-11T22:39:50.675 INFO:tasks.workunit.client.0.smithi059.stdout: "seq": 167503724554 2020-03-11T22:39:50.675 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:39:50.675 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:39:50.675 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pg_by_state": [ 2020-03-11T22:39:50.675 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.675 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "active+undersized+degraded", 2020-03-11T22:39:50.675 INFO:tasks.workunit.client.0.smithi059.stdout: "num": 32 2020-03-11T22:39:50.676 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:39:50.676 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:39:50.676 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pg_by_osd": [ 2020-03-11T22:39:50.676 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.676 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 2, 2020-03-11T22:39:50.676 INFO:tasks.workunit.client.0.smithi059.stdout: "num_primary_pg": 0, 2020-03-11T22:39:50.676 INFO:tasks.workunit.client.0.smithi059.stdout: "num_acting_pg": 32, 2020-03-11T22:39:50.676 INFO:tasks.workunit.client.0.smithi059.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:39:50.677 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.677 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.677 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 1, 2020-03-11T22:39:50.677 INFO:tasks.workunit.client.0.smithi059.stdout: "num_primary_pg": 0, 2020-03-11T22:39:50.677 INFO:tasks.workunit.client.0.smithi059.stdout: "num_acting_pg": 0, 2020-03-11T22:39:50.677 INFO:tasks.workunit.client.0.smithi059.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:39:50.677 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.678 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.678 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 0, 2020-03-11T22:39:50.678 INFO:tasks.workunit.client.0.smithi059.stdout: "num_primary_pg": 32, 2020-03-11T22:39:50.678 INFO:tasks.workunit.client.0.smithi059.stdout: "num_acting_pg": 32, 2020-03-11T22:39:50.678 INFO:tasks.workunit.client.0.smithi059.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:39:50.678 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:39:50.678 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:39:50.679 INFO:tasks.workunit.client.0.smithi059.stdout: "purged_snaps": [ 2020-03-11T22:39:50.679 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:39:50.679 INFO:tasks.workunit.client.0.smithi059.stdout: "pool": 1, 2020-03-11T22:39:50.679 INFO:tasks.workunit.client.0.smithi059.stdout: "purged_snaps": {} 2020-03-11T22:39:50.679 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:39:50.679 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:39:50.679 INFO:tasks.workunit.client.0.smithi059.stdout: "servicemap": { 2020-03-11T22:39:50.679 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": 1, 2020-03-11T22:39:50.680 INFO:tasks.workunit.client.0.smithi059.stdout: "modified": "2020-03-11T22:38:08.591885+0000", 2020-03-11T22:39:50.680 INFO:tasks.workunit.client.0.smithi059.stdout: "services": {} 2020-03-11T22:39:50.680 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:39:50.680 INFO:tasks.workunit.client.0.smithi059.stdout: "paxos": { 2020-03-11T22:39:50.680 INFO:tasks.workunit.client.0.smithi059.stdout: "first_committed": 1, 2020-03-11T22:39:50.680 INFO:tasks.workunit.client.0.smithi059.stdout: "last_committed": 316, 2020-03-11T22:39:50.680 INFO:tasks.workunit.client.0.smithi059.stdout: "last_pn": 0, 2020-03-11T22:39:50.680 INFO:tasks.workunit.client.0.smithi059.stdout: "accepted_pn": 0 2020-03-11T22:39:50.681 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:39:50.681 INFO:tasks.workunit.client.0.smithi059.stdout:} 2020-03-11T22:39:50.682 INFO:tasks.workunit.client.0.smithi059.stderr:report 222294428 2020-03-11T22:39:50.682 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: return 1 2020-03-11T22:39:50.682 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:738: TEST_divergent_3: expr 2 + 2 2020-03-11T22:39:50.682 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:738: TEST_divergent_3: objname=existing_4 2020-03-11T22:39:50.682 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:739: TEST_divergent_3: echo 'writing non-divergent object existing_4' 2020-03-11T22:39:50.683 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:740: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:39:50.683 INFO:tasks.workunit.client.0.smithi059.stdout:writing non-divergent object existing_4 2020-03-11T22:39:50.832 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:39:50.832 INFO:tasks.workunit.client.0.smithi059.stdout:1.1f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.448084+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:39:50.832 INFO:tasks.workunit.client.0.smithi059.stdout:1.1e 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.446855+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:39:50.832 INFO:tasks.workunit.client.0.smithi059.stdout:1.1d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.449135+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:39:50.832 INFO:tasks.workunit.client.0.smithi059.stdout:1.1c 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.446971+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:39:50.833 INFO:tasks.workunit.client.0.smithi059.stdout:1.1b 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.446984+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:39:50.833 INFO:tasks.workunit.client.0.smithi059.stdout:1.1a 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.444150+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:39:50.833 INFO:tasks.workunit.client.0.smithi059.stdout:1.19 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.444719+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:39:50.833 INFO:tasks.workunit.client.0.smithi059.stdout:1.18 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.448692+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:39:50.833 INFO:tasks.workunit.client.0.smithi059.stdout:1.17 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.446465+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:39:50.833 INFO:tasks.workunit.client.0.smithi059.stdout:1.16 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.446471+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:39:50.833 INFO:tasks.workunit.client.0.smithi059.stdout:1.15 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.445232+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:39:50.834 INFO:tasks.workunit.client.0.smithi059.stdout:1.14 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.445238+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:39:50.834 INFO:tasks.workunit.client.0.smithi059.stdout:1.13 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.447934+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:39:50.834 INFO:tasks.workunit.client.0.smithi059.stdout:1.12 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.445962+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:39:50.834 INFO:tasks.workunit.client.0.smithi059.stdout:1.11 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.445944+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:39:50.834 INFO:tasks.workunit.client.0.smithi059.stdout:1.10 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.445755+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:39:50.834 INFO:tasks.workunit.client.0.smithi059.stdout:1.5 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.445787+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:39:50.834 INFO:tasks.workunit.client.0.smithi059.stdout:1.4 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.447374+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:39:50.835 INFO:tasks.workunit.client.0.smithi059.stdout:1.3 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.444964+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:39:50.835 INFO:tasks.workunit.client.0.smithi059.stdout:1.2 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.445406+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:39:50.835 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:30.918910+0000 31'100 44:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:39:50.835 INFO:tasks.workunit.client.0.smithi059.stdout:1.1 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.446245+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:39:50.835 INFO:tasks.workunit.client.0.smithi059.stdout:1.6 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.447557+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:39:50.835 INFO:tasks.workunit.client.0.smithi059.stdout:1.7 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.444141+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:39:50.835 INFO:tasks.workunit.client.0.smithi059.stdout:1.8 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.444152+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:39:50.836 INFO:tasks.workunit.client.0.smithi059.stdout:1.9 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.444388+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:39:50.836 INFO:tasks.workunit.client.0.smithi059.stdout:1.a 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.447345+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:39:50.836 INFO:tasks.workunit.client.0.smithi059.stdout:1.b 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.444733+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:39:50.836 INFO:tasks.workunit.client.0.smithi059.stdout:1.c 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.444867+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:39:50.836 INFO:tasks.workunit.client.0.smithi059.stdout:1.d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.445468+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:39:50.836 INFO:tasks.workunit.client.0.smithi059.stdout:1.e 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.445193+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:39:50.836 INFO:tasks.workunit.client.0.smithi059.stdout:1.f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.446311+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:39:50.836 INFO:tasks.workunit.client.0.smithi059.stdout: 2020-03-11T22:39:50.837 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:39:50.837 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2020-03-11T22:39:50.841 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:741: TEST_divergent_3: rados -p test put existing_4 /etc/resolv.conf 2020-03-11T22:39:50.924 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:743: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:39:50.924 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:743: TEST_divergent_3: wait_for_clean 2020-03-11T22:39:50.925 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:39:50.925 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:39:50.925 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:39:50.925 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:39:50.925 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:39:50.926 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:39:50.926 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:39:50.926 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:39:50.926 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:39:50.926 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:39:50.984 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:39:50.984 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:39:50.985 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:39:50.985 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:39:50.985 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:39:51.347 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:39:51.348 INFO:tasks.workunit.client.0.smithi059.stderr:1 2020-03-11T22:39:51.348 INFO:tasks.workunit.client.0.smithi059.stderr:2' 2020-03-11T22:39:51.348 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:39:51.348 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:39:51.348 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:39:51.516 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=158913789965 2020-03-11T22:39:51.517 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-158913789965' 2020-03-11T22:39:51.517 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:39:51.517 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:39:51.612 INFO:tasks.workunit.client.0.smithi059.stderr:Error ENXIO: problem getting command descriptions from osd.1 2020-03-11T22:39:51.620 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq= 2020-03-11T22:39:51.620 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-158913789965 1-' 2020-03-11T22:39:51.621 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:39:51.621 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:39:51.757 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=167503724556 2020-03-11T22:39:51.757 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-158913789965 1- 2-167503724556' 2020-03-11T22:39:51.757 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:39:51.758 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-158913789965 2020-03-11T22:39:51.758 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:39:51.758 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:39:51.759 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-158913789965 2020-03-11T22:39:51.759 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:39:51.760 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=158913789965 2020-03-11T22:39:51.760 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 158913789965' 2020-03-11T22:39:51.760 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:39:51.760 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.0 seq 158913789965 2020-03-11T22:39:52.121 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 158913789963 -lt 158913789965 2020-03-11T22:39:52.121 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:39:53.122 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:39:53.123 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:39:53.501 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 158913789965 -lt 158913789965 2020-03-11T22:39:53.502 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:39:53.502 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1- 2020-03-11T22:39:53.502 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:39:53.503 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:39:53.503 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1- 2020-03-11T22:39:53.503 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:39:53.504 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq= 2020-03-11T22:39:53.504 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq ' 2020-03-11T22:39:53.505 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:39:53.505 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.1 seq 2020-03-11T22:39:53.925 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672975 -lt 2020-03-11T22:39:53.925 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 2079: test: 42949672975: unary operator expected 2020-03-11T22:39:53.925 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:39:53.926 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-167503724556 2020-03-11T22:39:53.926 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:39:53.928 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:39:53.928 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-167503724556 2020-03-11T22:39:53.928 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:39:53.928 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=167503724556 2020-03-11T22:39:53.929 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 167503724556' 2020-03-11T22:39:53.929 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:39:53.929 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.2 seq 167503724556 2020-03-11T22:39:54.292 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 167503724556 -lt 167503724556 2020-03-11T22:39:54.293 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:39:54.293 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:39:54.293 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:39:54.780 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 32 == 0 2020-03-11T22:39:54.780 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:39:54.780 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:39:54.780 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:39:54.780 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:39:54.780 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:39:54.781 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:39:54.781 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:39:55.131 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:39:55.132 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:39:55.132 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:39:55.132 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:39:55.607 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:39:55.607 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:39:55.607 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:39:55.607 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:39:55.608 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:39:55.608 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:39:55.708 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:39:55.708 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:39:55.708 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:39:55.709 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:39:55.709 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:39:55.709 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:39:55.709 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:39:55.709 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:39:56.061 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:39:56.062 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:39:56.062 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:39:56.062 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:39:56.533 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:39:56.533 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:39:56.533 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:39:56.533 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:39:56.534 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:39:56.534 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:39:56.534 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:39:56.534 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:39:56.534 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:39:57.037 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:39:57.038 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:39:57.038 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 1 >= 8 )) 2020-03-11T22:39:57.038 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:39:57.038 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.2 2020-03-11T22:39:57.239 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:39:57.240 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:39:57.240 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:39:57.240 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:39:57.240 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:39:57.241 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:39:57.241 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:39:57.241 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:39:57.597 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:39:57.598 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:39:57.598 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:39:57.598 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:39:58.079 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:39:58.079 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:39:58.080 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:39:58.080 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:39:58.080 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:39:58.080 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:39:58.080 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:39:58.080 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:39:58.081 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:39:58.552 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:39:58.552 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:39:58.552 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 2 >= 8 )) 2020-03-11T22:39:58.553 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:39:58.553 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.4 2020-03-11T22:39:59.165 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:39:59.165 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:39:59.166 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:39:59.166 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:39:59.166 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:39:59.166 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:39:59.166 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:39:59.168 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:39:59.316 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:39:59.316 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:39:59.316 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:39:59.318 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:39:59.787 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:39:59.787 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:39:59.787 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:39:59.788 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:39:59.788 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:39:59.788 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:39:59.788 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:39:59.788 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:39:59.788 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:40:00.291 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:40:00.291 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:40:00.291 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 3 >= 8 )) 2020-03-11T22:40:00.291 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:40:00.291 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.8 2020-03-11T22:40:01.093 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:40:01.093 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:40:01.093 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:40:01.093 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:40:01.093 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:40:01.093 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:40:01.094 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:40:01.094 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:40:01.459 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:40:01.460 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:40:01.460 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:40:01.460 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:40:01.930 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:40:01.930 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:40:01.930 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:40:01.930 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:40:01.931 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:40:01.931 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:40:01.931 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:40:01.931 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:40:01.931 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:40:02.405 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:40:02.405 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:40:02.405 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 4 >= 8 )) 2020-03-11T22:40:02.405 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:40:02.405 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 1.6 2020-03-11T22:40:04.006 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:40:04.006 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:40:04.007 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:40:04.007 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:40:04.007 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:40:04.007 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:40:04.007 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:40:04.007 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:40:04.377 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:40:04.378 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:40:04.378 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:40:04.378 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:40:04.848 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:40:04.848 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:40:04.848 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:40:04.849 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:40:04.849 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:40:04.849 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:40:04.849 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:40:04.849 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:40:04.849 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:40:05.319 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:40:05.320 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:40:05.320 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 5 >= 8 )) 2020-03-11T22:40:05.320 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:40:05.320 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 3.2 2020-03-11T22:40:08.520 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:40:08.520 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:40:08.521 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:40:08.521 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:40:08.522 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:40:08.522 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:40:08.522 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:40:08.522 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:40:08.888 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:40:08.889 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:40:08.889 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:40:08.889 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:40:09.357 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:40:09.357 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:40:09.358 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:40:09.358 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:40:09.358 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:40:09.358 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:40:09.358 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:40:09.358 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:40:09.358 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:40:09.834 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:40:09.834 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:40:09.834 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 6 >= 8 )) 2020-03-11T22:40:09.834 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:40:09.834 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 6.4 2020-03-11T22:40:16.235 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:40:16.235 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:40:16.236 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:40:16.236 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:40:16.236 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:40:16.236 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:40:16.236 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:40:16.236 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:40:16.607 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:40:16.608 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:40:16.608 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:40:16.608 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:40:17.087 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:40:17.087 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:40:17.087 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:40:17.088 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:40:17.088 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:40:17.088 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:40:17.088 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:40:17.088 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:40:17.088 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:40:17.562 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:40:17.563 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:40:17.563 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 7 >= 8 )) 2020-03-11T22:40:17.563 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:40:17.563 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 7.3 2020-03-11T22:40:24.866 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:40:24.866 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:40:24.866 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:40:24.866 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:40:24.866 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:40:24.866 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:40:24.867 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:40:24.867 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:40:25.262 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:40:25.263 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:40:25.263 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:40:25.263 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:40:25.736 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:40:25.737 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:40:25.737 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:40:25.737 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:40:25.737 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:40:25.737 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:40:25.737 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:40:25.738 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:40:25.738 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:40:26.212 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:40:26.213 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:40:26.213 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 8 >= 8 )) 2020-03-11T22:40:26.213 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: ceph report 2020-03-11T22:40:26.674 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2020-03-11T22:40:26.674 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_fingerprint": "4d1ca1d1-97bf-4599-82c8-e8c345704e49", 2020-03-11T22:40:26.674 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "15.1.0-1941-g4acc112", 2020-03-11T22:40:26.675 INFO:tasks.workunit.client.0.smithi059.stdout: "commit": "4acc1121e47039a3cdf2978ae458c10d153aa141", 2020-03-11T22:40:26.675 INFO:tasks.workunit.client.0.smithi059.stdout: "timestamp": "2020-03-11T22:40:26.664765+0000", 2020-03-11T22:40:26.675 INFO:tasks.workunit.client.0.smithi059.stdout: "tag": "", 2020-03-11T22:40:26.675 INFO:tasks.workunit.client.0.smithi059.stdout: "health": { 2020-03-11T22:40:26.675 INFO:tasks.workunit.client.0.smithi059.stdout: "status": "HEALTH_WARN", 2020-03-11T22:40:26.676 INFO:tasks.workunit.client.0.smithi059.stdout: "checks": { 2020-03-11T22:40:26.676 INFO:tasks.workunit.client.0.smithi059.stdout: "OSDMAP_FLAGS": { 2020-03-11T22:40:26.676 INFO:tasks.workunit.client.0.smithi059.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:40:26.676 INFO:tasks.workunit.client.0.smithi059.stdout: "summary": { 2020-03-11T22:40:26.677 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "nodown,noout,noin flag(s) set", 2020-03-11T22:40:26.677 INFO:tasks.workunit.client.0.smithi059.stdout: "count": 17 2020-03-11T22:40:26.677 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.677 INFO:tasks.workunit.client.0.smithi059.stdout: "detail": [], 2020-03-11T22:40:26.678 INFO:tasks.workunit.client.0.smithi059.stdout: "muted": false 2020-03-11T22:40:26.678 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.678 INFO:tasks.workunit.client.0.smithi059.stdout: "OSD_DOWN": { 2020-03-11T22:40:26.678 INFO:tasks.workunit.client.0.smithi059.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:40:26.679 INFO:tasks.workunit.client.0.smithi059.stdout: "summary": { 2020-03-11T22:40:26.679 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "1 osds down", 2020-03-11T22:40:26.679 INFO:tasks.workunit.client.0.smithi059.stdout: "count": 1 2020-03-11T22:40:26.679 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.680 INFO:tasks.workunit.client.0.smithi059.stdout: "detail": [ 2020-03-11T22:40:26.680 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.680 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "osd.1 (root=default,host=smithi059) is down" 2020-03-11T22:40:26.680 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:40:26.680 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:40:26.681 INFO:tasks.workunit.client.0.smithi059.stdout: "muted": false 2020-03-11T22:40:26.681 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.681 INFO:tasks.workunit.client.0.smithi059.stdout: "PG_DEGRADED": { 2020-03-11T22:40:26.681 INFO:tasks.workunit.client.0.smithi059.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:40:26.682 INFO:tasks.workunit.client.0.smithi059.stdout: "summary": { 2020-03-11T22:40:26.682 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 32 pgs degraded, 32 pgs undersized", 2020-03-11T22:40:26.682 INFO:tasks.workunit.client.0.smithi059.stdout: "count": 64 2020-03-11T22:40:26.682 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.682 INFO:tasks.workunit.client.0.smithi059.stdout: "detail": [ 2020-03-11T22:40:26.683 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.683 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "pg 1.0 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:40:26.683 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.683 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.684 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "pg 1.1 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:40:26.684 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.684 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.685 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "pg 1.2 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:40:26.685 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.685 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.686 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "pg 1.3 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:40:26.686 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.686 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.686 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "pg 1.4 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:40:26.687 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.687 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.687 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "pg 1.5 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:40:26.687 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.687 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.688 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "pg 1.6 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:40:26.688 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.688 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.688 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "pg 1.7 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:40:26.689 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.689 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.689 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "pg 1.8 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:40:26.690 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.690 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.690 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "pg 1.9 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:40:26.690 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.690 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.691 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "pg 1.a is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:40:26.691 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.691 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.691 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "pg 1.b is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:40:26.692 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.692 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.692 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "pg 1.c is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:40:26.692 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.693 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.693 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "pg 1.d is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:40:26.693 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.693 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.693 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "pg 1.e is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:40:26.694 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.694 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.694 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "pg 1.f is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:40:26.694 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.695 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.695 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "pg 1.10 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:40:26.695 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.695 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.696 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "pg 1.11 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:40:26.696 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.696 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.696 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "pg 1.12 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:40:26.696 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.697 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.697 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "pg 1.13 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:40:26.697 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.698 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.698 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "pg 1.14 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:40:26.698 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.698 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.699 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "pg 1.15 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:40:26.699 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.699 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.700 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "pg 1.16 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:40:26.700 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.700 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.700 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "pg 1.17 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:40:26.700 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.701 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.701 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "pg 1.18 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:40:26.701 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.701 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.702 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "pg 1.19 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:40:26.702 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.702 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.703 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "pg 1.1a is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:40:26.703 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.703 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.705 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "pg 1.1b is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:40:26.705 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.705 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.706 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "pg 1.1c is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:40:26.706 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.706 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.706 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "pg 1.1d is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:40:26.707 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.707 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.707 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "pg 1.1e is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:40:26.707 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.708 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.708 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "pg 1.1f is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:40:26.708 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:40:26.708 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:40:26.709 INFO:tasks.workunit.client.0.smithi059.stdout: "muted": false 2020-03-11T22:40:26.709 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.709 INFO:tasks.workunit.client.0.smithi059.stdout: "POOL_APP_NOT_ENABLED": { 2020-03-11T22:40:26.710 INFO:tasks.workunit.client.0.smithi059.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:40:26.710 INFO:tasks.workunit.client.0.smithi059.stdout: "summary": { 2020-03-11T22:40:26.710 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "1 pool(s) do not have an application enabled", 2020-03-11T22:40:26.710 INFO:tasks.workunit.client.0.smithi059.stdout: "count": 1 2020-03-11T22:40:26.711 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.711 INFO:tasks.workunit.client.0.smithi059.stdout: "detail": [ 2020-03-11T22:40:26.711 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.711 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "application not enabled on pool 'test'" 2020-03-11T22:40:26.711 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.712 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.712 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2020-03-11T22:40:26.712 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:40:26.712 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:40:26.712 INFO:tasks.workunit.client.0.smithi059.stdout: "muted": false 2020-03-11T22:40:26.712 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:40:26.713 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.713 INFO:tasks.workunit.client.0.smithi059.stdout: "mutes": [] 2020-03-11T22:40:26.713 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.713 INFO:tasks.workunit.client.0.smithi059.stdout: "monmap_first_committed": 1, 2020-03-11T22:40:26.714 INFO:tasks.workunit.client.0.smithi059.stdout: "monmap_last_committed": 1, 2020-03-11T22:40:26.714 INFO:tasks.workunit.client.0.smithi059.stdout: "monmap": { 2020-03-11T22:40:26.714 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": 1, 2020-03-11T22:40:26.714 INFO:tasks.workunit.client.0.smithi059.stdout: "fsid": "cb7f2e42-1934-4145-a3c3-29bdd93f8adb", 2020-03-11T22:40:26.714 INFO:tasks.workunit.client.0.smithi059.stdout: "modified": "2020-03-11T22:38:07.922938Z", 2020-03-11T22:40:26.714 INFO:tasks.workunit.client.0.smithi059.stdout: "created": "2020-03-11T22:38:07.922938Z", 2020-03-11T22:40:26.714 INFO:tasks.workunit.client.0.smithi059.stdout: "min_mon_release": 15, 2020-03-11T22:40:26.715 INFO:tasks.workunit.client.0.smithi059.stdout: "min_mon_release_name": "octopus", 2020-03-11T22:40:26.715 INFO:tasks.workunit.client.0.smithi059.stdout: "features": { 2020-03-11T22:40:26.715 INFO:tasks.workunit.client.0.smithi059.stdout: "persistent": [ 2020-03-11T22:40:26.715 INFO:tasks.workunit.client.0.smithi059.stdout: "kraken", 2020-03-11T22:40:26.716 INFO:tasks.workunit.client.0.smithi059.stdout: "luminous", 2020-03-11T22:40:26.716 INFO:tasks.workunit.client.0.smithi059.stdout: "mimic", 2020-03-11T22:40:26.716 INFO:tasks.workunit.client.0.smithi059.stdout: "osdmap-prune", 2020-03-11T22:40:26.716 INFO:tasks.workunit.client.0.smithi059.stdout: "nautilus", 2020-03-11T22:40:26.716 INFO:tasks.workunit.client.0.smithi059.stdout: "octopus" 2020-03-11T22:40:26.716 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:40:26.717 INFO:tasks.workunit.client.0.smithi059.stdout: "optional": [] 2020-03-11T22:40:26.717 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.717 INFO:tasks.workunit.client.0.smithi059.stdout: "mons": [ 2020-03-11T22:40:26.717 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.717 INFO:tasks.workunit.client.0.smithi059.stdout: "rank": 0, 2020-03-11T22:40:26.718 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "a", 2020-03-11T22:40:26.718 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addrs": { 2020-03-11T22:40:26.718 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:40:26.718 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.718 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:40:26.719 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:7115", 2020-03-11T22:40:26.719 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 0 2020-03-11T22:40:26.719 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:40:26.719 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:40:26.720 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.720 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:7115/0", 2020-03-11T22:40:26.720 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addr": "127.0.0.1:7115/0", 2020-03-11T22:40:26.720 INFO:tasks.workunit.client.0.smithi059.stdout: "priority": 0, 2020-03-11T22:40:26.720 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 0 2020-03-11T22:40:26.721 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:40:26.721 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:40:26.721 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.721 INFO:tasks.workunit.client.0.smithi059.stdout: "quorum": [ 2020-03-11T22:40:26.721 INFO:tasks.workunit.client.0.smithi059.stdout: 0 2020-03-11T22:40:26.722 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:40:26.722 INFO:tasks.workunit.client.0.smithi059.stdout: "osdmap": { 2020-03-11T22:40:26.722 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": 44, 2020-03-11T22:40:26.723 INFO:tasks.workunit.client.0.smithi059.stdout: "fsid": "cb7f2e42-1934-4145-a3c3-29bdd93f8adb", 2020-03-11T22:40:26.723 INFO:tasks.workunit.client.0.smithi059.stdout: "created": "2020-03-11T22:38:08.591539+0000", 2020-03-11T22:40:26.723 INFO:tasks.workunit.client.0.smithi059.stdout: "modified": "2020-03-11T22:39:15.166894+0000", 2020-03-11T22:40:26.723 INFO:tasks.workunit.client.0.smithi059.stdout: "last_up_change": "2020-03-11T22:39:13.304952+0000", 2020-03-11T22:40:26.723 INFO:tasks.workunit.client.0.smithi059.stdout: "last_in_change": "2020-03-11T22:39:14.457686+0000", 2020-03-11T22:40:26.723 INFO:tasks.workunit.client.0.smithi059.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-03-11T22:40:26.724 INFO:tasks.workunit.client.0.smithi059.stdout: "flags_num": 5800384, 2020-03-11T22:40:26.724 INFO:tasks.workunit.client.0.smithi059.stdout: "flags_set": [ 2020-03-11T22:40:26.724 INFO:tasks.workunit.client.0.smithi059.stdout: "nodown", 2020-03-11T22:40:26.724 INFO:tasks.workunit.client.0.smithi059.stdout: "noin", 2020-03-11T22:40:26.724 INFO:tasks.workunit.client.0.smithi059.stdout: "noout", 2020-03-11T22:40:26.725 INFO:tasks.workunit.client.0.smithi059.stdout: "pglog_hardlimit", 2020-03-11T22:40:26.725 INFO:tasks.workunit.client.0.smithi059.stdout: "purged_snapdirs", 2020-03-11T22:40:26.725 INFO:tasks.workunit.client.0.smithi059.stdout: "recovery_deletes", 2020-03-11T22:40:26.725 INFO:tasks.workunit.client.0.smithi059.stdout: "sortbitwise" 2020-03-11T22:40:26.726 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:40:26.726 INFO:tasks.workunit.client.0.smithi059.stdout: "crush_version": 7, 2020-03-11T22:40:26.726 INFO:tasks.workunit.client.0.smithi059.stdout: "full_ratio": 0.99000000953674316, 2020-03-11T22:40:26.726 INFO:tasks.workunit.client.0.smithi059.stdout: "backfillfull_ratio": 0.99000000953674316, 2020-03-11T22:40:26.727 INFO:tasks.workunit.client.0.smithi059.stdout: "nearfull_ratio": 0.85000002384185791, 2020-03-11T22:40:26.727 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_snapshot": "", 2020-03-11T22:40:26.727 INFO:tasks.workunit.client.0.smithi059.stdout: "pool_max": 1, 2020-03-11T22:40:26.727 INFO:tasks.workunit.client.0.smithi059.stdout: "max_osd": 3, 2020-03-11T22:40:26.727 INFO:tasks.workunit.client.0.smithi059.stdout: "require_min_compat_client": "jewel", 2020-03-11T22:40:26.728 INFO:tasks.workunit.client.0.smithi059.stdout: "min_compat_client": "jewel", 2020-03-11T22:40:26.728 INFO:tasks.workunit.client.0.smithi059.stdout: "require_osd_release": "octopus", 2020-03-11T22:40:26.728 INFO:tasks.workunit.client.0.smithi059.stdout: "pools": [ 2020-03-11T22:40:26.728 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.728 INFO:tasks.workunit.client.0.smithi059.stdout: "pool": 1, 2020-03-11T22:40:26.728 INFO:tasks.workunit.client.0.smithi059.stdout: "pool_name": "test", 2020-03-11T22:40:26.729 INFO:tasks.workunit.client.0.smithi059.stdout: "create_time": "2020-03-11T22:38:30.749350+0000", 2020-03-11T22:40:26.729 INFO:tasks.workunit.client.0.smithi059.stdout: "flags": 1, 2020-03-11T22:40:26.729 INFO:tasks.workunit.client.0.smithi059.stdout: "flags_names": "hashpspool", 2020-03-11T22:40:26.729 INFO:tasks.workunit.client.0.smithi059.stdout: "type": 1, 2020-03-11T22:40:26.730 INFO:tasks.workunit.client.0.smithi059.stdout: "size": 3, 2020-03-11T22:40:26.730 INFO:tasks.workunit.client.0.smithi059.stdout: "min_size": 2, 2020-03-11T22:40:26.730 INFO:tasks.workunit.client.0.smithi059.stdout: "crush_rule": 0, 2020-03-11T22:40:26.730 INFO:tasks.workunit.client.0.smithi059.stdout: "object_hash": 2, 2020-03-11T22:40:26.730 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_autoscale_mode": "on", 2020-03-11T22:40:26.731 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_num": 32, 2020-03-11T22:40:26.731 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_placement_num": 1, 2020-03-11T22:40:26.731 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_placement_num_target": 32, 2020-03-11T22:40:26.731 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_num_target": 32, 2020-03-11T22:40:26.731 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_num_pending": 32, 2020-03-11T22:40:26.732 INFO:tasks.workunit.client.0.smithi059.stdout: "last_pg_merge_meta": { 2020-03-11T22:40:26.732 INFO:tasks.workunit.client.0.smithi059.stdout: "source_pgid": "0.0", 2020-03-11T22:40:26.732 INFO:tasks.workunit.client.0.smithi059.stdout: "ready_epoch": 0, 2020-03-11T22:40:26.732 INFO:tasks.workunit.client.0.smithi059.stdout: "last_epoch_started": 0, 2020-03-11T22:40:26.732 INFO:tasks.workunit.client.0.smithi059.stdout: "last_epoch_clean": 0, 2020-03-11T22:40:26.732 INFO:tasks.workunit.client.0.smithi059.stdout: "source_version": "0'0", 2020-03-11T22:40:26.733 INFO:tasks.workunit.client.0.smithi059.stdout: "target_version": "0'0" 2020-03-11T22:40:26.733 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.733 INFO:tasks.workunit.client.0.smithi059.stdout: "last_change": "43", 2020-03-11T22:40:26.733 INFO:tasks.workunit.client.0.smithi059.stdout: "last_force_op_resend": "0", 2020-03-11T22:40:26.733 INFO:tasks.workunit.client.0.smithi059.stdout: "last_force_op_resend_prenautilus": "0", 2020-03-11T22:40:26.733 INFO:tasks.workunit.client.0.smithi059.stdout: "last_force_op_resend_preluminous": "43", 2020-03-11T22:40:26.734 INFO:tasks.workunit.client.0.smithi059.stdout: "auid": 0, 2020-03-11T22:40:26.734 INFO:tasks.workunit.client.0.smithi059.stdout: "snap_mode": "selfmanaged", 2020-03-11T22:40:26.734 INFO:tasks.workunit.client.0.smithi059.stdout: "snap_seq": 0, 2020-03-11T22:40:26.734 INFO:tasks.workunit.client.0.smithi059.stdout: "snap_epoch": 0, 2020-03-11T22:40:26.734 INFO:tasks.workunit.client.0.smithi059.stdout: "pool_snaps": [], 2020-03-11T22:40:26.734 INFO:tasks.workunit.client.0.smithi059.stdout: "removed_snaps": "[]", 2020-03-11T22:40:26.734 INFO:tasks.workunit.client.0.smithi059.stdout: "quota_max_bytes": 0, 2020-03-11T22:40:26.734 INFO:tasks.workunit.client.0.smithi059.stdout: "quota_max_objects": 0, 2020-03-11T22:40:26.735 INFO:tasks.workunit.client.0.smithi059.stdout: "tiers": [], 2020-03-11T22:40:26.735 INFO:tasks.workunit.client.0.smithi059.stdout: "tier_of": -1, 2020-03-11T22:40:26.735 INFO:tasks.workunit.client.0.smithi059.stdout: "read_tier": -1, 2020-03-11T22:40:26.735 INFO:tasks.workunit.client.0.smithi059.stdout: "write_tier": -1, 2020-03-11T22:40:26.735 INFO:tasks.workunit.client.0.smithi059.stdout: "cache_mode": "none", 2020-03-11T22:40:26.735 INFO:tasks.workunit.client.0.smithi059.stdout: "target_max_bytes": 0, 2020-03-11T22:40:26.735 INFO:tasks.workunit.client.0.smithi059.stdout: "target_max_objects": 0, 2020-03-11T22:40:26.736 INFO:tasks.workunit.client.0.smithi059.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-03-11T22:40:26.736 INFO:tasks.workunit.client.0.smithi059.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-03-11T22:40:26.736 INFO:tasks.workunit.client.0.smithi059.stdout: "cache_target_full_ratio_micro": 800000, 2020-03-11T22:40:26.736 INFO:tasks.workunit.client.0.smithi059.stdout: "cache_min_flush_age": 0, 2020-03-11T22:40:26.736 INFO:tasks.workunit.client.0.smithi059.stdout: "cache_min_evict_age": 0, 2020-03-11T22:40:26.736 INFO:tasks.workunit.client.0.smithi059.stdout: "erasure_code_profile": "", 2020-03-11T22:40:26.736 INFO:tasks.workunit.client.0.smithi059.stdout: "hit_set_params": { 2020-03-11T22:40:26.737 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "none" 2020-03-11T22:40:26.737 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.737 INFO:tasks.workunit.client.0.smithi059.stdout: "hit_set_period": 0, 2020-03-11T22:40:26.737 INFO:tasks.workunit.client.0.smithi059.stdout: "hit_set_count": 0, 2020-03-11T22:40:26.737 INFO:tasks.workunit.client.0.smithi059.stdout: "use_gmt_hitset": true, 2020-03-11T22:40:26.737 INFO:tasks.workunit.client.0.smithi059.stdout: "min_read_recency_for_promote": 0, 2020-03-11T22:40:26.737 INFO:tasks.workunit.client.0.smithi059.stdout: "min_write_recency_for_promote": 0, 2020-03-11T22:40:26.738 INFO:tasks.workunit.client.0.smithi059.stdout: "hit_set_grade_decay_rate": 0, 2020-03-11T22:40:26.738 INFO:tasks.workunit.client.0.smithi059.stdout: "hit_set_search_last_n": 0, 2020-03-11T22:40:26.738 INFO:tasks.workunit.client.0.smithi059.stdout: "grade_table": [], 2020-03-11T22:40:26.738 INFO:tasks.workunit.client.0.smithi059.stdout: "stripe_width": 0, 2020-03-11T22:40:26.738 INFO:tasks.workunit.client.0.smithi059.stdout: "expected_num_objects": 0, 2020-03-11T22:40:26.738 INFO:tasks.workunit.client.0.smithi059.stdout: "fast_read": false, 2020-03-11T22:40:26.738 INFO:tasks.workunit.client.0.smithi059.stdout: "options": {}, 2020-03-11T22:40:26.738 INFO:tasks.workunit.client.0.smithi059.stdout: "application_metadata": {} 2020-03-11T22:40:26.739 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:40:26.739 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:40:26.739 INFO:tasks.workunit.client.0.smithi059.stdout: "osds": [ 2020-03-11T22:40:26.739 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.739 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 0, 2020-03-11T22:40:26.739 INFO:tasks.workunit.client.0.smithi059.stdout: "uuid": "9e248648-07e4-4309-92d7-2283b32a88ea", 2020-03-11T22:40:26.740 INFO:tasks.workunit.client.0.smithi059.stdout: "up": 1, 2020-03-11T22:40:26.740 INFO:tasks.workunit.client.0.smithi059.stdout: "in": 1, 2020-03-11T22:40:26.740 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 1, 2020-03-11T22:40:26.740 INFO:tasks.workunit.client.0.smithi059.stdout: "primary_affinity": 1, 2020-03-11T22:40:26.740 INFO:tasks.workunit.client.0.smithi059.stdout: "last_clean_begin": 5, 2020-03-11T22:40:26.740 INFO:tasks.workunit.client.0.smithi059.stdout: "last_clean_end": 31, 2020-03-11T22:40:26.740 INFO:tasks.workunit.client.0.smithi059.stdout: "up_from": 37, 2020-03-11T22:40:26.741 INFO:tasks.workunit.client.0.smithi059.stdout: "up_thru": 43, 2020-03-11T22:40:26.741 INFO:tasks.workunit.client.0.smithi059.stdout: "down_at": 32, 2020-03-11T22:40:26.741 INFO:tasks.workunit.client.0.smithi059.stdout: "lost_at": 0, 2020-03-11T22:40:26.741 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addrs": { 2020-03-11T22:40:26.741 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:40:26.741 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.741 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:40:26.741 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6802", 2020-03-11T22:40:26.742 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 39814 2020-03-11T22:40:26.742 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.742 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.742 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:40:26.742 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6803", 2020-03-11T22:40:26.742 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 39814 2020-03-11T22:40:26.742 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:40:26.743 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:40:26.743 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.743 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_addrs": { 2020-03-11T22:40:26.743 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:40:26.743 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.743 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:40:26.743 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6804", 2020-03-11T22:40:26.744 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 39814 2020-03-11T22:40:26.744 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.744 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.744 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:40:26.744 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6805", 2020-03-11T22:40:26.744 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 39814 2020-03-11T22:40:26.744 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:40:26.744 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:40:26.745 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.745 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_back_addrs": { 2020-03-11T22:40:26.745 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:40:26.745 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.745 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:40:26.745 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6808", 2020-03-11T22:40:26.745 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 39814 2020-03-11T22:40:26.746 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.746 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.746 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:40:26.746 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6809", 2020-03-11T22:40:26.746 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 39814 2020-03-11T22:40:26.746 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:40:26.746 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:40:26.747 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.747 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_front_addrs": { 2020-03-11T22:40:26.747 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:40:26.747 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.747 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:40:26.747 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6806", 2020-03-11T22:40:26.747 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 39814 2020-03-11T22:40:26.747 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.748 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.748 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:40:26.748 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6807", 2020-03-11T22:40:26.748 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 39814 2020-03-11T22:40:26.748 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:40:26.748 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:40:26.749 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.749 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addr": "127.0.0.1:6803/39814", 2020-03-11T22:40:26.749 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_addr": "127.0.0.1:6805/39814", 2020-03-11T22:40:26.749 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_back_addr": "127.0.0.1:6809/39814", 2020-03-11T22:40:26.749 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_front_addr": "127.0.0.1:6807/39814", 2020-03-11T22:40:26.749 INFO:tasks.workunit.client.0.smithi059.stdout: "state": [ 2020-03-11T22:40:26.749 INFO:tasks.workunit.client.0.smithi059.stdout: "exists", 2020-03-11T22:40:26.750 INFO:tasks.workunit.client.0.smithi059.stdout: "up" 2020-03-11T22:40:26.750 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:40:26.750 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.750 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.750 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 1, 2020-03-11T22:40:26.750 INFO:tasks.workunit.client.0.smithi059.stdout: "uuid": "98cb8180-5961-424c-a790-9139892a46ae", 2020-03-11T22:40:26.750 INFO:tasks.workunit.client.0.smithi059.stdout: "up": 0, 2020-03-11T22:40:26.750 INFO:tasks.workunit.client.0.smithi059.stdout: "in": 1, 2020-03-11T22:40:26.751 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 1, 2020-03-11T22:40:26.751 INFO:tasks.workunit.client.0.smithi059.stdout: "primary_affinity": 1, 2020-03-11T22:40:26.751 INFO:tasks.workunit.client.0.smithi059.stdout: "last_clean_begin": 0, 2020-03-11T22:40:26.751 INFO:tasks.workunit.client.0.smithi059.stdout: "last_clean_end": 0, 2020-03-11T22:40:26.751 INFO:tasks.workunit.client.0.smithi059.stdout: "up_from": 10, 2020-03-11T22:40:26.751 INFO:tasks.workunit.client.0.smithi059.stdout: "up_thru": 23, 2020-03-11T22:40:26.751 INFO:tasks.workunit.client.0.smithi059.stdout: "down_at": 33, 2020-03-11T22:40:26.752 INFO:tasks.workunit.client.0.smithi059.stdout: "lost_at": 0, 2020-03-11T22:40:26.752 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addrs": { 2020-03-11T22:40:26.752 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:40:26.752 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.752 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:40:26.752 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:40:26.752 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 35398 2020-03-11T22:40:26.753 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.753 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.753 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:40:26.753 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:40:26.753 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 35398 2020-03-11T22:40:26.753 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:40:26.753 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:40:26.754 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.755 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_addrs": { 2020-03-11T22:40:26.755 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:40:26.755 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.756 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:40:26.756 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:40:26.756 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 35398 2020-03-11T22:40:26.756 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.756 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.756 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:40:26.756 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:40:26.757 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 35398 2020-03-11T22:40:26.757 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:40:26.757 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:40:26.757 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.757 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_back_addrs": { 2020-03-11T22:40:26.757 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:40:26.757 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.758 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:40:26.758 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:40:26.758 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 35398 2020-03-11T22:40:26.758 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.758 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.758 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:40:26.758 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:40:26.759 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 35398 2020-03-11T22:40:26.759 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:40:26.759 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:40:26.759 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.759 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_front_addrs": { 2020-03-11T22:40:26.759 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:40:26.759 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.760 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:40:26.760 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:40:26.760 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 35398 2020-03-11T22:40:26.760 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.760 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.760 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:40:26.760 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:40:26.761 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 35398 2020-03-11T22:40:26.761 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:40:26.761 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:40:26.761 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.761 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addr": "127.0.0.1:6811/35398", 2020-03-11T22:40:26.761 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_addr": "127.0.0.1:6813/35398", 2020-03-11T22:40:26.761 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_back_addr": "127.0.0.1:6817/35398", 2020-03-11T22:40:26.762 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_front_addr": "127.0.0.1:6815/35398", 2020-03-11T22:40:26.762 INFO:tasks.workunit.client.0.smithi059.stdout: "state": [ 2020-03-11T22:40:26.762 INFO:tasks.workunit.client.0.smithi059.stdout: "exists" 2020-03-11T22:40:26.762 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:40:26.762 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.762 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.762 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 2, 2020-03-11T22:40:26.762 INFO:tasks.workunit.client.0.smithi059.stdout: "uuid": "18b16e96-77ce-45af-9ce0-23852cb42c25", 2020-03-11T22:40:26.763 INFO:tasks.workunit.client.0.smithi059.stdout: "up": 1, 2020-03-11T22:40:26.763 INFO:tasks.workunit.client.0.smithi059.stdout: "in": 1, 2020-03-11T22:40:26.763 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 1, 2020-03-11T22:40:26.763 INFO:tasks.workunit.client.0.smithi059.stdout: "primary_affinity": 1, 2020-03-11T22:40:26.763 INFO:tasks.workunit.client.0.smithi059.stdout: "last_clean_begin": 15, 2020-03-11T22:40:26.763 INFO:tasks.workunit.client.0.smithi059.stdout: "last_clean_end": 31, 2020-03-11T22:40:26.763 INFO:tasks.workunit.client.0.smithi059.stdout: "up_from": 39, 2020-03-11T22:40:26.764 INFO:tasks.workunit.client.0.smithi059.stdout: "up_thru": 0, 2020-03-11T22:40:26.764 INFO:tasks.workunit.client.0.smithi059.stdout: "down_at": 34, 2020-03-11T22:40:26.764 INFO:tasks.workunit.client.0.smithi059.stdout: "lost_at": 0, 2020-03-11T22:40:26.764 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addrs": { 2020-03-11T22:40:26.764 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:40:26.764 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.764 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:40:26.765 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:40:26.765 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 40008 2020-03-11T22:40:26.765 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.765 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.765 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:40:26.765 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:40:26.765 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 40008 2020-03-11T22:40:26.765 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:40:26.766 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:40:26.766 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.766 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_addrs": { 2020-03-11T22:40:26.766 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:40:26.766 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.766 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:40:26.767 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:40:26.767 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 40008 2020-03-11T22:40:26.767 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.767 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.767 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:40:26.767 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:40:26.767 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 40008 2020-03-11T22:40:26.768 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:40:26.768 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:40:26.768 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.768 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_back_addrs": { 2020-03-11T22:40:26.768 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:40:26.768 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.768 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:40:26.768 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:40:26.769 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 40008 2020-03-11T22:40:26.769 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.769 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.769 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:40:26.769 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:40:26.769 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 40008 2020-03-11T22:40:26.769 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:40:26.770 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:40:26.770 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.770 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_front_addrs": { 2020-03-11T22:40:26.770 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:40:26.770 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.770 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:40:26.770 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:40:26.771 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 40008 2020-03-11T22:40:26.771 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.771 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.771 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:40:26.771 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:40:26.771 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 40008 2020-03-11T22:40:26.771 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:40:26.771 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:40:26.772 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.772 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addr": "127.0.0.1:6811/40008", 2020-03-11T22:40:26.772 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_addr": "127.0.0.1:6813/40008", 2020-03-11T22:40:26.772 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_back_addr": "127.0.0.1:6817/40008", 2020-03-11T22:40:26.772 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_front_addr": "127.0.0.1:6815/40008", 2020-03-11T22:40:26.772 INFO:tasks.workunit.client.0.smithi059.stdout: "state": [ 2020-03-11T22:40:26.773 INFO:tasks.workunit.client.0.smithi059.stdout: "exists", 2020-03-11T22:40:26.773 INFO:tasks.workunit.client.0.smithi059.stdout: "up" 2020-03-11T22:40:26.773 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:40:26.773 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:40:26.773 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:40:26.773 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_xinfo": [ 2020-03-11T22:40:26.773 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.774 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 0, 2020-03-11T22:40:26.774 INFO:tasks.workunit.client.0.smithi059.stdout: "down_stamp": "2020-03-11T22:39:06.563045+0000", 2020-03-11T22:40:26.774 INFO:tasks.workunit.client.0.smithi059.stdout: "laggy_probability": 0, 2020-03-11T22:40:26.774 INFO:tasks.workunit.client.0.smithi059.stdout: "laggy_interval": 0, 2020-03-11T22:40:26.774 INFO:tasks.workunit.client.0.smithi059.stdout: "features": 4540138292836696063, 2020-03-11T22:40:26.774 INFO:tasks.workunit.client.0.smithi059.stdout: "old_weight": 0, 2020-03-11T22:40:26.774 INFO:tasks.workunit.client.0.smithi059.stdout: "last_purged_snaps_scrub": "2020-03-11T22:38:15.567036+0000", 2020-03-11T22:40:26.775 INFO:tasks.workunit.client.0.smithi059.stdout: "dead_epoch": 0 2020-03-11T22:40:26.775 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.775 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.775 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 1, 2020-03-11T22:40:26.775 INFO:tasks.workunit.client.0.smithi059.stdout: "down_stamp": "2020-03-11T22:39:07.122111+0000", 2020-03-11T22:40:26.775 INFO:tasks.workunit.client.0.smithi059.stdout: "laggy_probability": 0, 2020-03-11T22:40:26.775 INFO:tasks.workunit.client.0.smithi059.stdout: "laggy_interval": 0, 2020-03-11T22:40:26.775 INFO:tasks.workunit.client.0.smithi059.stdout: "features": 4540138292836696063, 2020-03-11T22:40:26.776 INFO:tasks.workunit.client.0.smithi059.stdout: "old_weight": 0, 2020-03-11T22:40:26.776 INFO:tasks.workunit.client.0.smithi059.stdout: "last_purged_snaps_scrub": "2020-03-11T22:38:21.900248+0000", 2020-03-11T22:40:26.776 INFO:tasks.workunit.client.0.smithi059.stdout: "dead_epoch": 0 2020-03-11T22:40:26.776 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.776 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.776 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 2, 2020-03-11T22:40:26.776 INFO:tasks.workunit.client.0.smithi059.stdout: "down_stamp": "2020-03-11T22:39:07.637108+0000", 2020-03-11T22:40:26.777 INFO:tasks.workunit.client.0.smithi059.stdout: "laggy_probability": 0, 2020-03-11T22:40:26.777 INFO:tasks.workunit.client.0.smithi059.stdout: "laggy_interval": 0, 2020-03-11T22:40:26.777 INFO:tasks.workunit.client.0.smithi059.stdout: "features": 4540138292836696063, 2020-03-11T22:40:26.777 INFO:tasks.workunit.client.0.smithi059.stdout: "old_weight": 0, 2020-03-11T22:40:26.777 INFO:tasks.workunit.client.0.smithi059.stdout: "last_purged_snaps_scrub": "2020-03-11T22:38:28.351224+0000", 2020-03-11T22:40:26.778 INFO:tasks.workunit.client.0.smithi059.stdout: "dead_epoch": 0 2020-03-11T22:40:26.778 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:40:26.778 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:40:26.778 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_upmap": [], 2020-03-11T22:40:26.778 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_upmap_items": [], 2020-03-11T22:40:26.778 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_temp": [], 2020-03-11T22:40:26.778 INFO:tasks.workunit.client.0.smithi059.stdout: "primary_temp": [], 2020-03-11T22:40:26.779 INFO:tasks.workunit.client.0.smithi059.stdout: "blacklist": {}, 2020-03-11T22:40:26.779 INFO:tasks.workunit.client.0.smithi059.stdout: "erasure_code_profiles": { 2020-03-11T22:40:26.779 INFO:tasks.workunit.client.0.smithi059.stdout: "default": { 2020-03-11T22:40:26.779 INFO:tasks.workunit.client.0.smithi059.stdout: "k": "2", 2020-03-11T22:40:26.779 INFO:tasks.workunit.client.0.smithi059.stdout: "m": "2", 2020-03-11T22:40:26.779 INFO:tasks.workunit.client.0.smithi059.stdout: "plugin": "jerasure", 2020-03-11T22:40:26.779 INFO:tasks.workunit.client.0.smithi059.stdout: "technique": "reed_sol_van" 2020-03-11T22:40:26.779 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:40:26.780 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.780 INFO:tasks.workunit.client.0.smithi059.stdout: "removed_snaps_queue": [], 2020-03-11T22:40:26.780 INFO:tasks.workunit.client.0.smithi059.stdout: "new_removed_snaps": [], 2020-03-11T22:40:26.780 INFO:tasks.workunit.client.0.smithi059.stdout: "new_purged_snaps": [], 2020-03-11T22:40:26.780 INFO:tasks.workunit.client.0.smithi059.stdout: "crush_node_flags": {}, 2020-03-11T22:40:26.780 INFO:tasks.workunit.client.0.smithi059.stdout: "device_class_flags": {} 2020-03-11T22:40:26.780 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.781 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_metadata": [ 2020-03-11T22:40:26.781 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.781 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 0, 2020-03-11T22:40:26.781 INFO:tasks.workunit.client.0.smithi059.stdout: "arch": "x86_64", 2020-03-11T22:40:26.781 INFO:tasks.workunit.client.0.smithi059.stdout: "back_addr": "[v2:127.0.0.1:6804/39814,v1:127.0.0.1:6805/39814]", 2020-03-11T22:40:26.781 INFO:tasks.workunit.client.0.smithi059.stdout: "back_iface": "", 2020-03-11T22:40:26.781 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs": "1", 2020-03-11T22:40:26.782 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:40:26.782 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:40:26.782 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:40:26.783 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:40:26.783 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:40:26.783 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:40:26.783 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:40:26.784 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2020-03-11T22:40:26.784 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:40:26.784 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:40:26.784 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:40:26.784 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:40:26.785 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_release": "octopus", 2020-03-11T22:40:26.785 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:40:26.785 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:40:26.785 INFO:tasks.workunit.client.0.smithi059.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-03-11T22:40:26.786 INFO:tasks.workunit.client.0.smithi059.stdout: "default_device_class": "hdd", 2020-03-11T22:40:26.786 INFO:tasks.workunit.client.0.smithi059.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4J2P7", 2020-03-11T22:40:26.786 INFO:tasks.workunit.client.0.smithi059.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:40:26.786 INFO:tasks.workunit.client.0.smithi059.stdout: "devices": "sda", 2020-03-11T22:40:26.787 INFO:tasks.workunit.client.0.smithi059.stdout: "distro": "centos", 2020-03-11T22:40:26.787 INFO:tasks.workunit.client.0.smithi059.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:40:26.787 INFO:tasks.workunit.client.0.smithi059.stdout: "distro_version": "8", 2020-03-11T22:40:26.787 INFO:tasks.workunit.client.0.smithi059.stdout: "front_addr": "[v2:127.0.0.1:6802/39814,v1:127.0.0.1:6803/39814]", 2020-03-11T22:40:26.788 INFO:tasks.workunit.client.0.smithi059.stdout: "front_iface": "", 2020-03-11T22:40:26.788 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/39814,v1:127.0.0.1:6809/39814]", 2020-03-11T22:40:26.788 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/39814,v1:127.0.0.1:6807/39814]", 2020-03-11T22:40:26.788 INFO:tasks.workunit.client.0.smithi059.stdout: "hostname": "smithi059", 2020-03-11T22:40:26.788 INFO:tasks.workunit.client.0.smithi059.stdout: "journal_rotational": "1", 2020-03-11T22:40:26.790 INFO:tasks.workunit.client.0.smithi059.stdout: "kernel_description": "#1 SMP Wed Feb 5 02:00:39 UTC 2020", 2020-03-11T22:40:26.790 INFO:tasks.workunit.client.0.smithi059.stdout: "kernel_version": "4.18.0-147.5.1.el8_1.x86_64", 2020-03-11T22:40:26.790 INFO:tasks.workunit.client.0.smithi059.stdout: "mem_swap_kb": "0", 2020-03-11T22:40:26.790 INFO:tasks.workunit.client.0.smithi059.stdout: "mem_total_kb": "32662084", 2020-03-11T22:40:26.791 INFO:tasks.workunit.client.0.smithi059.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:40:26.791 INFO:tasks.workunit.client.0.smithi059.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:40:26.791 INFO:tasks.workunit.client.0.smithi059.stdout: "os": "Linux", 2020-03-11T22:40:26.791 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_data": "td/divergent-priors/0", 2020-03-11T22:40:26.791 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:40:26.792 INFO:tasks.workunit.client.0.smithi059.stdout: "rotational": "1" 2020-03-11T22:40:26.792 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.792 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.792 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 1, 2020-03-11T22:40:26.793 INFO:tasks.workunit.client.0.smithi059.stdout: "arch": "x86_64", 2020-03-11T22:40:26.793 INFO:tasks.workunit.client.0.smithi059.stdout: "back_addr": "[v2:127.0.0.1:6812/35398,v1:127.0.0.1:6813/35398]", 2020-03-11T22:40:26.793 INFO:tasks.workunit.client.0.smithi059.stdout: "back_iface": "", 2020-03-11T22:40:26.793 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs": "1", 2020-03-11T22:40:26.793 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:40:26.794 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:40:26.794 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:40:26.794 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:40:26.794 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:40:26.794 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:40:26.795 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:40:26.795 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2020-03-11T22:40:26.795 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:40:26.795 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:40:26.796 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:40:26.796 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:40:26.796 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_release": "octopus", 2020-03-11T22:40:26.796 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:40:26.796 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:40:26.797 INFO:tasks.workunit.client.0.smithi059.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-03-11T22:40:26.797 INFO:tasks.workunit.client.0.smithi059.stdout: "default_device_class": "hdd", 2020-03-11T22:40:26.797 INFO:tasks.workunit.client.0.smithi059.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4J2P7", 2020-03-11T22:40:26.797 INFO:tasks.workunit.client.0.smithi059.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:40:26.798 INFO:tasks.workunit.client.0.smithi059.stdout: "devices": "sda", 2020-03-11T22:40:26.798 INFO:tasks.workunit.client.0.smithi059.stdout: "distro": "centos", 2020-03-11T22:40:26.798 INFO:tasks.workunit.client.0.smithi059.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:40:26.798 INFO:tasks.workunit.client.0.smithi059.stdout: "distro_version": "8", 2020-03-11T22:40:26.799 INFO:tasks.workunit.client.0.smithi059.stdout: "front_addr": "[v2:127.0.0.1:6810/35398,v1:127.0.0.1:6811/35398]", 2020-03-11T22:40:26.799 INFO:tasks.workunit.client.0.smithi059.stdout: "front_iface": "", 2020-03-11T22:40:26.799 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/35398,v1:127.0.0.1:6817/35398]", 2020-03-11T22:40:26.799 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/35398,v1:127.0.0.1:6815/35398]", 2020-03-11T22:40:26.799 INFO:tasks.workunit.client.0.smithi059.stdout: "hostname": "smithi059", 2020-03-11T22:40:26.800 INFO:tasks.workunit.client.0.smithi059.stdout: "journal_rotational": "1", 2020-03-11T22:40:26.800 INFO:tasks.workunit.client.0.smithi059.stdout: "kernel_description": "#1 SMP Wed Feb 5 02:00:39 UTC 2020", 2020-03-11T22:40:26.800 INFO:tasks.workunit.client.0.smithi059.stdout: "kernel_version": "4.18.0-147.5.1.el8_1.x86_64", 2020-03-11T22:40:26.800 INFO:tasks.workunit.client.0.smithi059.stdout: "mem_swap_kb": "0", 2020-03-11T22:40:26.800 INFO:tasks.workunit.client.0.smithi059.stdout: "mem_total_kb": "32662084", 2020-03-11T22:40:26.801 INFO:tasks.workunit.client.0.smithi059.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:40:26.801 INFO:tasks.workunit.client.0.smithi059.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:40:26.801 INFO:tasks.workunit.client.0.smithi059.stdout: "os": "Linux", 2020-03-11T22:40:26.801 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_data": "td/divergent-priors/1", 2020-03-11T22:40:26.801 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:40:26.802 INFO:tasks.workunit.client.0.smithi059.stdout: "rotational": "1" 2020-03-11T22:40:26.802 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.802 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.802 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 2, 2020-03-11T22:40:26.802 INFO:tasks.workunit.client.0.smithi059.stdout: "arch": "x86_64", 2020-03-11T22:40:26.803 INFO:tasks.workunit.client.0.smithi059.stdout: "back_addr": "[v2:127.0.0.1:6812/40008,v1:127.0.0.1:6813/40008]", 2020-03-11T22:40:26.803 INFO:tasks.workunit.client.0.smithi059.stdout: "back_iface": "", 2020-03-11T22:40:26.803 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs": "1", 2020-03-11T22:40:26.803 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:40:26.804 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:40:26.804 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:40:26.804 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:40:26.804 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:40:26.804 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:40:26.805 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:40:26.805 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2020-03-11T22:40:26.805 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:40:26.805 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:40:26.805 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:40:26.806 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:40:26.806 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_release": "octopus", 2020-03-11T22:40:26.806 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:40:26.807 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:40:26.807 INFO:tasks.workunit.client.0.smithi059.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-03-11T22:40:26.807 INFO:tasks.workunit.client.0.smithi059.stdout: "default_device_class": "hdd", 2020-03-11T22:40:26.807 INFO:tasks.workunit.client.0.smithi059.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4J2P7", 2020-03-11T22:40:26.808 INFO:tasks.workunit.client.0.smithi059.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:40:26.808 INFO:tasks.workunit.client.0.smithi059.stdout: "devices": "sda", 2020-03-11T22:40:26.808 INFO:tasks.workunit.client.0.smithi059.stdout: "distro": "centos", 2020-03-11T22:40:26.808 INFO:tasks.workunit.client.0.smithi059.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:40:26.808 INFO:tasks.workunit.client.0.smithi059.stdout: "distro_version": "8", 2020-03-11T22:40:26.809 INFO:tasks.workunit.client.0.smithi059.stdout: "front_addr": "[v2:127.0.0.1:6810/40008,v1:127.0.0.1:6811/40008]", 2020-03-11T22:40:26.809 INFO:tasks.workunit.client.0.smithi059.stdout: "front_iface": "", 2020-03-11T22:40:26.809 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/40008,v1:127.0.0.1:6817/40008]", 2020-03-11T22:40:26.809 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/40008,v1:127.0.0.1:6815/40008]", 2020-03-11T22:40:26.809 INFO:tasks.workunit.client.0.smithi059.stdout: "hostname": "smithi059", 2020-03-11T22:40:26.810 INFO:tasks.workunit.client.0.smithi059.stdout: "journal_rotational": "1", 2020-03-11T22:40:26.810 INFO:tasks.workunit.client.0.smithi059.stdout: "kernel_description": "#1 SMP Wed Feb 5 02:00:39 UTC 2020", 2020-03-11T22:40:26.810 INFO:tasks.workunit.client.0.smithi059.stdout: "kernel_version": "4.18.0-147.5.1.el8_1.x86_64", 2020-03-11T22:40:26.810 INFO:tasks.workunit.client.0.smithi059.stdout: "mem_swap_kb": "0", 2020-03-11T22:40:26.811 INFO:tasks.workunit.client.0.smithi059.stdout: "mem_total_kb": "32662084", 2020-03-11T22:40:26.811 INFO:tasks.workunit.client.0.smithi059.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:40:26.811 INFO:tasks.workunit.client.0.smithi059.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:40:26.811 INFO:tasks.workunit.client.0.smithi059.stdout: "os": "Linux", 2020-03-11T22:40:26.811 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_data": "td/divergent-priors/2", 2020-03-11T22:40:26.812 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:40:26.812 INFO:tasks.workunit.client.0.smithi059.stdout: "rotational": "1" 2020-03-11T22:40:26.812 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:40:26.812 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:40:26.813 INFO:tasks.workunit.client.0.smithi059.stdout: "osdmap_first_committed": 1, 2020-03-11T22:40:26.813 INFO:tasks.workunit.client.0.smithi059.stdout: "osdmap_last_committed": 44, 2020-03-11T22:40:26.813 INFO:tasks.workunit.client.0.smithi059.stdout: "crushmap": { 2020-03-11T22:40:26.813 INFO:tasks.workunit.client.0.smithi059.stdout: "devices": [ 2020-03-11T22:40:26.813 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.814 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 0, 2020-03-11T22:40:26.814 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "osd.0", 2020-03-11T22:40:26.814 INFO:tasks.workunit.client.0.smithi059.stdout: "class": "hdd" 2020-03-11T22:40:26.814 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.814 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.815 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 1, 2020-03-11T22:40:26.815 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "osd.1", 2020-03-11T22:40:26.815 INFO:tasks.workunit.client.0.smithi059.stdout: "class": "hdd" 2020-03-11T22:40:26.815 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.815 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.815 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 2, 2020-03-11T22:40:26.815 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "osd.2", 2020-03-11T22:40:26.815 INFO:tasks.workunit.client.0.smithi059.stdout: "class": "hdd" 2020-03-11T22:40:26.816 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:40:26.816 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:40:26.816 INFO:tasks.workunit.client.0.smithi059.stdout: "types": [ 2020-03-11T22:40:26.816 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.816 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 0, 2020-03-11T22:40:26.816 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "osd" 2020-03-11T22:40:26.816 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.817 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.817 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 1, 2020-03-11T22:40:26.817 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "host" 2020-03-11T22:40:26.817 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.817 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.817 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 2, 2020-03-11T22:40:26.817 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "chassis" 2020-03-11T22:40:26.818 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.818 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.818 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 3, 2020-03-11T22:40:26.818 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "rack" 2020-03-11T22:40:26.818 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.818 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.818 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 4, 2020-03-11T22:40:26.819 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "row" 2020-03-11T22:40:26.819 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.819 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.819 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 5, 2020-03-11T22:40:26.819 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "pdu" 2020-03-11T22:40:26.819 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.819 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.819 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 6, 2020-03-11T22:40:26.820 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "pod" 2020-03-11T22:40:26.820 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.820 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.820 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 7, 2020-03-11T22:40:26.820 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "room" 2020-03-11T22:40:26.820 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.821 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.821 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 8, 2020-03-11T22:40:26.821 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "datacenter" 2020-03-11T22:40:26.821 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.821 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.821 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 9, 2020-03-11T22:40:26.821 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "zone" 2020-03-11T22:40:26.822 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.822 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.822 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 10, 2020-03-11T22:40:26.822 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "region" 2020-03-11T22:40:26.822 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.822 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.822 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 11, 2020-03-11T22:40:26.822 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "root" 2020-03-11T22:40:26.823 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:40:26.823 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:40:26.823 INFO:tasks.workunit.client.0.smithi059.stdout: "buckets": [ 2020-03-11T22:40:26.823 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.823 INFO:tasks.workunit.client.0.smithi059.stdout: "id": -1, 2020-03-11T22:40:26.823 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "default", 2020-03-11T22:40:26.823 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 11, 2020-03-11T22:40:26.824 INFO:tasks.workunit.client.0.smithi059.stdout: "type_name": "root", 2020-03-11T22:40:26.824 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 19206, 2020-03-11T22:40:26.824 INFO:tasks.workunit.client.0.smithi059.stdout: "alg": "straw2", 2020-03-11T22:40:26.824 INFO:tasks.workunit.client.0.smithi059.stdout: "hash": "rjenkins1", 2020-03-11T22:40:26.825 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2020-03-11T22:40:26.825 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.825 INFO:tasks.workunit.client.0.smithi059.stdout: "id": -3, 2020-03-11T22:40:26.826 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 19206, 2020-03-11T22:40:26.826 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 0 2020-03-11T22:40:26.826 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:40:26.826 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:40:26.826 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.827 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.827 INFO:tasks.workunit.client.0.smithi059.stdout: "id": -2, 2020-03-11T22:40:26.827 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "default~hdd", 2020-03-11T22:40:26.827 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 11, 2020-03-11T22:40:26.828 INFO:tasks.workunit.client.0.smithi059.stdout: "type_name": "root", 2020-03-11T22:40:26.828 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 19206, 2020-03-11T22:40:26.828 INFO:tasks.workunit.client.0.smithi059.stdout: "alg": "straw2", 2020-03-11T22:40:26.828 INFO:tasks.workunit.client.0.smithi059.stdout: "hash": "rjenkins1", 2020-03-11T22:40:26.829 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2020-03-11T22:40:26.829 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.829 INFO:tasks.workunit.client.0.smithi059.stdout: "id": -4, 2020-03-11T22:40:26.829 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 19206, 2020-03-11T22:40:26.830 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 0 2020-03-11T22:40:26.830 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:40:26.830 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:40:26.830 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.830 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.831 INFO:tasks.workunit.client.0.smithi059.stdout: "id": -3, 2020-03-11T22:40:26.831 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "smithi059", 2020-03-11T22:40:26.831 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 1, 2020-03-11T22:40:26.831 INFO:tasks.workunit.client.0.smithi059.stdout: "type_name": "host", 2020-03-11T22:40:26.832 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 19206, 2020-03-11T22:40:26.832 INFO:tasks.workunit.client.0.smithi059.stdout: "alg": "straw2", 2020-03-11T22:40:26.832 INFO:tasks.workunit.client.0.smithi059.stdout: "hash": "rjenkins1", 2020-03-11T22:40:26.832 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2020-03-11T22:40:26.833 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.833 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 0, 2020-03-11T22:40:26.833 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 6402, 2020-03-11T22:40:26.833 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 0 2020-03-11T22:40:26.834 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.834 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.834 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 1, 2020-03-11T22:40:26.834 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 6402, 2020-03-11T22:40:26.834 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 1 2020-03-11T22:40:26.835 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.835 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.835 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 2, 2020-03-11T22:40:26.835 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 6402, 2020-03-11T22:40:26.835 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 2 2020-03-11T22:40:26.836 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:40:26.836 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:40:26.836 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.836 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.836 INFO:tasks.workunit.client.0.smithi059.stdout: "id": -4, 2020-03-11T22:40:26.837 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "smithi059~hdd", 2020-03-11T22:40:26.837 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 1, 2020-03-11T22:40:26.837 INFO:tasks.workunit.client.0.smithi059.stdout: "type_name": "host", 2020-03-11T22:40:26.837 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 19206, 2020-03-11T22:40:26.838 INFO:tasks.workunit.client.0.smithi059.stdout: "alg": "straw2", 2020-03-11T22:40:26.838 INFO:tasks.workunit.client.0.smithi059.stdout: "hash": "rjenkins1", 2020-03-11T22:40:26.838 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2020-03-11T22:40:26.838 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.838 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 0, 2020-03-11T22:40:26.839 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 6402, 2020-03-11T22:40:26.839 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 0 2020-03-11T22:40:26.839 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.839 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.839 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 1, 2020-03-11T22:40:26.840 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 6402, 2020-03-11T22:40:26.840 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 1 2020-03-11T22:40:26.840 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.840 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.841 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 2, 2020-03-11T22:40:26.841 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 6402, 2020-03-11T22:40:26.841 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 2 2020-03-11T22:40:26.841 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:40:26.841 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:40:26.842 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:40:26.842 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:40:26.842 INFO:tasks.workunit.client.0.smithi059.stdout: "rules": [ 2020-03-11T22:40:26.842 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.842 INFO:tasks.workunit.client.0.smithi059.stdout: "rule_id": 0, 2020-03-11T22:40:26.843 INFO:tasks.workunit.client.0.smithi059.stdout: "rule_name": "replicated_rule", 2020-03-11T22:40:26.843 INFO:tasks.workunit.client.0.smithi059.stdout: "ruleset": 0, 2020-03-11T22:40:26.843 INFO:tasks.workunit.client.0.smithi059.stdout: "type": 1, 2020-03-11T22:40:26.843 INFO:tasks.workunit.client.0.smithi059.stdout: "min_size": 1, 2020-03-11T22:40:26.843 INFO:tasks.workunit.client.0.smithi059.stdout: "max_size": 10, 2020-03-11T22:40:26.844 INFO:tasks.workunit.client.0.smithi059.stdout: "steps": [ 2020-03-11T22:40:26.844 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.844 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "take", 2020-03-11T22:40:26.844 INFO:tasks.workunit.client.0.smithi059.stdout: "item": -1, 2020-03-11T22:40:26.844 INFO:tasks.workunit.client.0.smithi059.stdout: "item_name": "default" 2020-03-11T22:40:26.844 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.845 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.845 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "choose_firstn", 2020-03-11T22:40:26.845 INFO:tasks.workunit.client.0.smithi059.stdout: "num": 0, 2020-03-11T22:40:26.845 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "osd" 2020-03-11T22:40:26.845 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.845 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.846 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "emit" 2020-03-11T22:40:26.846 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:40:26.846 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:40:26.846 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:40:26.846 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:40:26.846 INFO:tasks.workunit.client.0.smithi059.stdout: "tunables": { 2020-03-11T22:40:26.846 INFO:tasks.workunit.client.0.smithi059.stdout: "choose_local_tries": 0, 2020-03-11T22:40:26.847 INFO:tasks.workunit.client.0.smithi059.stdout: "choose_local_fallback_tries": 0, 2020-03-11T22:40:26.847 INFO:tasks.workunit.client.0.smithi059.stdout: "choose_total_tries": 50, 2020-03-11T22:40:26.847 INFO:tasks.workunit.client.0.smithi059.stdout: "chooseleaf_descend_once": 1, 2020-03-11T22:40:26.847 INFO:tasks.workunit.client.0.smithi059.stdout: "chooseleaf_vary_r": 1, 2020-03-11T22:40:26.847 INFO:tasks.workunit.client.0.smithi059.stdout: "chooseleaf_stable": 1, 2020-03-11T22:40:26.847 INFO:tasks.workunit.client.0.smithi059.stdout: "straw_calc_version": 1, 2020-03-11T22:40:26.848 INFO:tasks.workunit.client.0.smithi059.stdout: "allowed_bucket_algs": 54, 2020-03-11T22:40:26.848 INFO:tasks.workunit.client.0.smithi059.stdout: "profile": "jewel", 2020-03-11T22:40:26.848 INFO:tasks.workunit.client.0.smithi059.stdout: "optimal_tunables": 1, 2020-03-11T22:40:26.848 INFO:tasks.workunit.client.0.smithi059.stdout: "legacy_tunables": 0, 2020-03-11T22:40:26.848 INFO:tasks.workunit.client.0.smithi059.stdout: "minimum_required_version": "jewel", 2020-03-11T22:40:26.848 INFO:tasks.workunit.client.0.smithi059.stdout: "require_feature_tunables": 1, 2020-03-11T22:40:26.848 INFO:tasks.workunit.client.0.smithi059.stdout: "require_feature_tunables2": 1, 2020-03-11T22:40:26.848 INFO:tasks.workunit.client.0.smithi059.stdout: "has_v2_rules": 0, 2020-03-11T22:40:26.849 INFO:tasks.workunit.client.0.smithi059.stdout: "require_feature_tunables3": 1, 2020-03-11T22:40:26.849 INFO:tasks.workunit.client.0.smithi059.stdout: "has_v3_rules": 0, 2020-03-11T22:40:26.849 INFO:tasks.workunit.client.0.smithi059.stdout: "has_v4_buckets": 1, 2020-03-11T22:40:26.849 INFO:tasks.workunit.client.0.smithi059.stdout: "require_feature_tunables5": 1, 2020-03-11T22:40:26.849 INFO:tasks.workunit.client.0.smithi059.stdout: "has_v5_rules": 0 2020-03-11T22:40:26.849 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.849 INFO:tasks.workunit.client.0.smithi059.stdout: "choose_args": {} 2020-03-11T22:40:26.850 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.850 INFO:tasks.workunit.client.0.smithi059.stdout: "fsmap": { 2020-03-11T22:40:26.850 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": 1, 2020-03-11T22:40:26.850 INFO:tasks.workunit.client.0.smithi059.stdout: "default_fscid": -1, 2020-03-11T22:40:26.850 INFO:tasks.workunit.client.0.smithi059.stdout: "compat": { 2020-03-11T22:40:26.850 INFO:tasks.workunit.client.0.smithi059.stdout: "compat": {}, 2020-03-11T22:40:26.850 INFO:tasks.workunit.client.0.smithi059.stdout: "ro_compat": {}, 2020-03-11T22:40:26.851 INFO:tasks.workunit.client.0.smithi059.stdout: "incompat": { 2020-03-11T22:40:26.851 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_1": "base v0.20", 2020-03-11T22:40:26.851 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_2": "client writeable ranges", 2020-03-11T22:40:26.851 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_3": "default file layouts on dirs", 2020-03-11T22:40:26.851 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_4": "dir inode in separate object", 2020-03-11T22:40:26.851 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_5": "mds uses versioned encoding", 2020-03-11T22:40:26.851 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_6": "dirfrag is stored in omap", 2020-03-11T22:40:26.852 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_8": "no anchor table", 2020-03-11T22:40:26.852 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_9": "file layout v2", 2020-03-11T22:40:26.852 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_10": "snaprealm v2" 2020-03-11T22:40:26.852 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:40:26.852 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.852 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_flags": { 2020-03-11T22:40:26.852 INFO:tasks.workunit.client.0.smithi059.stdout: "enable_multiple": false, 2020-03-11T22:40:26.852 INFO:tasks.workunit.client.0.smithi059.stdout: "ever_enabled_multiple": false 2020-03-11T22:40:26.853 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.853 INFO:tasks.workunit.client.0.smithi059.stdout: "standbys": [], 2020-03-11T22:40:26.853 INFO:tasks.workunit.client.0.smithi059.stdout: "filesystems": [] 2020-03-11T22:40:26.853 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.853 INFO:tasks.workunit.client.0.smithi059.stdout: "mdsmap_first_committed": 1, 2020-03-11T22:40:26.853 INFO:tasks.workunit.client.0.smithi059.stdout: "mdsmap_last_committed": 1, 2020-03-11T22:40:26.853 INFO:tasks.workunit.client.0.smithi059.stdout: "auth": { 2020-03-11T22:40:26.854 INFO:tasks.workunit.client.0.smithi059.stdout: "first_committed": 1, 2020-03-11T22:40:26.854 INFO:tasks.workunit.client.0.smithi059.stdout: "last_committed": 5, 2020-03-11T22:40:26.854 INFO:tasks.workunit.client.0.smithi059.stdout: "num_secrets": 3 2020-03-11T22:40:26.854 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.854 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pg": 32, 2020-03-11T22:40:26.854 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pg_active": 32, 2020-03-11T22:40:26.854 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pg_unknown": 0, 2020-03-11T22:40:26.855 INFO:tasks.workunit.client.0.smithi059.stdout: "num_osd": 3, 2020-03-11T22:40:26.855 INFO:tasks.workunit.client.0.smithi059.stdout: "pool_sum": { 2020-03-11T22:40:26.855 INFO:tasks.workunit.client.0.smithi059.stdout: "stat_sum": { 2020-03-11T22:40:26.855 INFO:tasks.workunit.client.0.smithi059.stdout: "num_bytes": 47947, 2020-03-11T22:40:26.855 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects": 100, 2020-03-11T22:40:26.855 INFO:tasks.workunit.client.0.smithi059.stdout: "num_object_clones": 0, 2020-03-11T22:40:26.855 INFO:tasks.workunit.client.0.smithi059.stdout: "num_object_copies": 300, 2020-03-11T22:40:26.855 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:40:26.856 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_missing": 0, 2020-03-11T22:40:26.856 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_degraded": 100, 2020-03-11T22:40:26.856 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_misplaced": 0, 2020-03-11T22:40:26.856 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_unfound": 0, 2020-03-11T22:40:26.856 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_dirty": 100, 2020-03-11T22:40:26.856 INFO:tasks.workunit.client.0.smithi059.stdout: "num_whiteouts": 0, 2020-03-11T22:40:26.856 INFO:tasks.workunit.client.0.smithi059.stdout: "num_read": 0, 2020-03-11T22:40:26.857 INFO:tasks.workunit.client.0.smithi059.stdout: "num_read_kb": 0, 2020-03-11T22:40:26.857 INFO:tasks.workunit.client.0.smithi059.stdout: "num_write": 101, 2020-03-11T22:40:26.857 INFO:tasks.workunit.client.0.smithi059.stdout: "num_write_kb": 101, 2020-03-11T22:40:26.857 INFO:tasks.workunit.client.0.smithi059.stdout: "num_scrub_errors": 0, 2020-03-11T22:40:26.857 INFO:tasks.workunit.client.0.smithi059.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:40:26.857 INFO:tasks.workunit.client.0.smithi059.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:40:26.857 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_recovered": 0, 2020-03-11T22:40:26.858 INFO:tasks.workunit.client.0.smithi059.stdout: "num_bytes_recovered": 0, 2020-03-11T22:40:26.858 INFO:tasks.workunit.client.0.smithi059.stdout: "num_keys_recovered": 0, 2020-03-11T22:40:26.858 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_omap": 0, 2020-03-11T22:40:26.858 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:40:26.858 INFO:tasks.workunit.client.0.smithi059.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:40:26.858 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush": 0, 2020-03-11T22:40:26.858 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush_kb": 0, 2020-03-11T22:40:26.858 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict": 0, 2020-03-11T22:40:26.859 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict_kb": 0, 2020-03-11T22:40:26.859 INFO:tasks.workunit.client.0.smithi059.stdout: "num_promote": 0, 2020-03-11T22:40:26.859 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush_mode_high": 0, 2020-03-11T22:40:26.859 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush_mode_low": 0, 2020-03-11T22:40:26.859 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict_mode_some": 0, 2020-03-11T22:40:26.859 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict_mode_full": 0, 2020-03-11T22:40:26.859 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_pinned": 0, 2020-03-11T22:40:26.860 INFO:tasks.workunit.client.0.smithi059.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:40:26.860 INFO:tasks.workunit.client.0.smithi059.stdout: "num_large_omap_objects": 0, 2020-03-11T22:40:26.860 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_manifest": 0, 2020-03-11T22:40:26.860 INFO:tasks.workunit.client.0.smithi059.stdout: "num_omap_bytes": 0, 2020-03-11T22:40:26.860 INFO:tasks.workunit.client.0.smithi059.stdout: "num_omap_keys": 0, 2020-03-11T22:40:26.860 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_repaired": 0 2020-03-11T22:40:26.860 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.861 INFO:tasks.workunit.client.0.smithi059.stdout: "store_stats": { 2020-03-11T22:40:26.861 INFO:tasks.workunit.client.0.smithi059.stdout: "total": 0, 2020-03-11T22:40:26.861 INFO:tasks.workunit.client.0.smithi059.stdout: "available": 0, 2020-03-11T22:40:26.861 INFO:tasks.workunit.client.0.smithi059.stdout: "internally_reserved": 0, 2020-03-11T22:40:26.861 INFO:tasks.workunit.client.0.smithi059.stdout: "allocated": 0, 2020-03-11T22:40:26.862 INFO:tasks.workunit.client.0.smithi059.stdout: "data_stored": 0, 2020-03-11T22:40:26.862 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed": 0, 2020-03-11T22:40:26.862 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_allocated": 0, 2020-03-11T22:40:26.862 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_original": 0, 2020-03-11T22:40:26.862 INFO:tasks.workunit.client.0.smithi059.stdout: "omap_allocated": 0, 2020-03-11T22:40:26.862 INFO:tasks.workunit.client.0.smithi059.stdout: "internal_metadata": 0 2020-03-11T22:40:26.862 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.863 INFO:tasks.workunit.client.0.smithi059.stdout: "log_size": 513, 2020-03-11T22:40:26.863 INFO:tasks.workunit.client.0.smithi059.stdout: "ondisk_log_size": 513, 2020-03-11T22:40:26.863 INFO:tasks.workunit.client.0.smithi059.stdout: "up": 64, 2020-03-11T22:40:26.863 INFO:tasks.workunit.client.0.smithi059.stdout: "acting": 64, 2020-03-11T22:40:26.863 INFO:tasks.workunit.client.0.smithi059.stdout: "num_store_stats": 0 2020-03-11T22:40:26.863 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.864 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_sum": { 2020-03-11T22:40:26.864 INFO:tasks.workunit.client.0.smithi059.stdout: "up_from": 0, 2020-03-11T22:40:26.864 INFO:tasks.workunit.client.0.smithi059.stdout: "seq": 0, 2020-03-11T22:40:26.864 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pgs": 65, 2020-03-11T22:40:26.864 INFO:tasks.workunit.client.0.smithi059.stdout: "num_osds": 3, 2020-03-11T22:40:26.864 INFO:tasks.workunit.client.0.smithi059.stdout: "num_per_pool_osds": 3, 2020-03-11T22:40:26.864 INFO:tasks.workunit.client.0.smithi059.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:40:26.865 INFO:tasks.workunit.client.0.smithi059.stdout: "kb": 314572800, 2020-03-11T22:40:26.865 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used": 3180672, 2020-03-11T22:40:26.865 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used_data": 34624, 2020-03-11T22:40:26.865 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used_omap": 7, 2020-03-11T22:40:26.865 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used_meta": 3145720, 2020-03-11T22:40:26.865 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_avail": 311392128, 2020-03-11T22:40:26.865 INFO:tasks.workunit.client.0.smithi059.stdout: "statfs": { 2020-03-11T22:40:26.866 INFO:tasks.workunit.client.0.smithi059.stdout: "total": 322122547200, 2020-03-11T22:40:26.866 INFO:tasks.workunit.client.0.smithi059.stdout: "available": 318865539072, 2020-03-11T22:40:26.866 INFO:tasks.workunit.client.0.smithi059.stdout: "internally_reserved": 3221225472, 2020-03-11T22:40:26.866 INFO:tasks.workunit.client.0.smithi059.stdout: "allocated": 35454976, 2020-03-11T22:40:26.866 INFO:tasks.workunit.client.0.smithi059.stdout: "data_stored": 437942, 2020-03-11T22:40:26.866 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed": 0, 2020-03-11T22:40:26.866 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_allocated": 0, 2020-03-11T22:40:26.867 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_original": 0, 2020-03-11T22:40:26.867 INFO:tasks.workunit.client.0.smithi059.stdout: "omap_allocated": 8190, 2020-03-11T22:40:26.867 INFO:tasks.workunit.client.0.smithi059.stdout: "internal_metadata": 3221217282 2020-03-11T22:40:26.867 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.867 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_peers": [], 2020-03-11T22:40:26.867 INFO:tasks.workunit.client.0.smithi059.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:40:26.867 INFO:tasks.workunit.client.0.smithi059.stdout: "num_snap_trimming": 0, 2020-03-11T22:40:26.867 INFO:tasks.workunit.client.0.smithi059.stdout: "num_shards_repaired": 0, 2020-03-11T22:40:26.868 INFO:tasks.workunit.client.0.smithi059.stdout: "op_queue_age_hist": { 2020-03-11T22:40:26.868 INFO:tasks.workunit.client.0.smithi059.stdout: "histogram": [], 2020-03-11T22:40:26.868 INFO:tasks.workunit.client.0.smithi059.stdout: "upper_bound": 1 2020-03-11T22:40:26.868 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.868 INFO:tasks.workunit.client.0.smithi059.stdout: "perf_stat": { 2020-03-11T22:40:26.868 INFO:tasks.workunit.client.0.smithi059.stdout: "commit_latency_ms": 0, 2020-03-11T22:40:26.868 INFO:tasks.workunit.client.0.smithi059.stdout: "apply_latency_ms": 0, 2020-03-11T22:40:26.869 INFO:tasks.workunit.client.0.smithi059.stdout: "commit_latency_ns": 0, 2020-03-11T22:40:26.869 INFO:tasks.workunit.client.0.smithi059.stdout: "apply_latency_ns": 0 2020-03-11T22:40:26.869 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.869 INFO:tasks.workunit.client.0.smithi059.stdout: "alerts": [], 2020-03-11T22:40:26.869 INFO:tasks.workunit.client.0.smithi059.stdout: "network_ping_times": [] 2020-03-11T22:40:26.869 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.869 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_sum_by_class": { 2020-03-11T22:40:26.869 INFO:tasks.workunit.client.0.smithi059.stdout: "hdd": { 2020-03-11T22:40:26.870 INFO:tasks.workunit.client.0.smithi059.stdout: "up_from": 0, 2020-03-11T22:40:26.870 INFO:tasks.workunit.client.0.smithi059.stdout: "seq": 0, 2020-03-11T22:40:26.870 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pgs": 65, 2020-03-11T22:40:26.870 INFO:tasks.workunit.client.0.smithi059.stdout: "num_osds": 3, 2020-03-11T22:40:26.870 INFO:tasks.workunit.client.0.smithi059.stdout: "num_per_pool_osds": 3, 2020-03-11T22:40:26.870 INFO:tasks.workunit.client.0.smithi059.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:40:26.870 INFO:tasks.workunit.client.0.smithi059.stdout: "kb": 314572800, 2020-03-11T22:40:26.871 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used": 3180672, 2020-03-11T22:40:26.871 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used_data": 34624, 2020-03-11T22:40:26.871 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used_omap": 7, 2020-03-11T22:40:26.871 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used_meta": 3145720, 2020-03-11T22:40:26.871 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_avail": 311392128, 2020-03-11T22:40:26.871 INFO:tasks.workunit.client.0.smithi059.stdout: "statfs": { 2020-03-11T22:40:26.871 INFO:tasks.workunit.client.0.smithi059.stdout: "total": 322122547200, 2020-03-11T22:40:26.872 INFO:tasks.workunit.client.0.smithi059.stdout: "available": 318865539072, 2020-03-11T22:40:26.872 INFO:tasks.workunit.client.0.smithi059.stdout: "internally_reserved": 3221225472, 2020-03-11T22:40:26.872 INFO:tasks.workunit.client.0.smithi059.stdout: "allocated": 35454976, 2020-03-11T22:40:26.872 INFO:tasks.workunit.client.0.smithi059.stdout: "data_stored": 437942, 2020-03-11T22:40:26.872 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed": 0, 2020-03-11T22:40:26.872 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_allocated": 0, 2020-03-11T22:40:26.872 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_original": 0, 2020-03-11T22:40:26.872 INFO:tasks.workunit.client.0.smithi059.stdout: "omap_allocated": 8190, 2020-03-11T22:40:26.873 INFO:tasks.workunit.client.0.smithi059.stdout: "internal_metadata": 3221217282 2020-03-11T22:40:26.873 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.873 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_peers": [], 2020-03-11T22:40:26.873 INFO:tasks.workunit.client.0.smithi059.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:40:26.873 INFO:tasks.workunit.client.0.smithi059.stdout: "num_snap_trimming": 0, 2020-03-11T22:40:26.873 INFO:tasks.workunit.client.0.smithi059.stdout: "num_shards_repaired": 0, 2020-03-11T22:40:26.873 INFO:tasks.workunit.client.0.smithi059.stdout: "op_queue_age_hist": { 2020-03-11T22:40:26.874 INFO:tasks.workunit.client.0.smithi059.stdout: "histogram": [], 2020-03-11T22:40:26.874 INFO:tasks.workunit.client.0.smithi059.stdout: "upper_bound": 1 2020-03-11T22:40:26.874 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.874 INFO:tasks.workunit.client.0.smithi059.stdout: "perf_stat": { 2020-03-11T22:40:26.874 INFO:tasks.workunit.client.0.smithi059.stdout: "commit_latency_ms": 0, 2020-03-11T22:40:26.874 INFO:tasks.workunit.client.0.smithi059.stdout: "apply_latency_ms": 0, 2020-03-11T22:40:26.874 INFO:tasks.workunit.client.0.smithi059.stdout: "commit_latency_ns": 0, 2020-03-11T22:40:26.874 INFO:tasks.workunit.client.0.smithi059.stdout: "apply_latency_ns": 0 2020-03-11T22:40:26.875 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.875 INFO:tasks.workunit.client.0.smithi059.stdout: "alerts": [], 2020-03-11T22:40:26.875 INFO:tasks.workunit.client.0.smithi059.stdout: "network_ping_times": [] 2020-03-11T22:40:26.875 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:40:26.875 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.875 INFO:tasks.workunit.client.0.smithi059.stdout: "pool_stats": [ 2020-03-11T22:40:26.875 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.876 INFO:tasks.workunit.client.0.smithi059.stdout: "poolid": 1, 2020-03-11T22:40:26.876 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pg": 32, 2020-03-11T22:40:26.876 INFO:tasks.workunit.client.0.smithi059.stdout: "stat_sum": { 2020-03-11T22:40:26.876 INFO:tasks.workunit.client.0.smithi059.stdout: "num_bytes": 47947, 2020-03-11T22:40:26.876 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects": 100, 2020-03-11T22:40:26.876 INFO:tasks.workunit.client.0.smithi059.stdout: "num_object_clones": 0, 2020-03-11T22:40:26.876 INFO:tasks.workunit.client.0.smithi059.stdout: "num_object_copies": 300, 2020-03-11T22:40:26.877 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:40:26.877 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_missing": 0, 2020-03-11T22:40:26.877 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_degraded": 100, 2020-03-11T22:40:26.877 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_misplaced": 0, 2020-03-11T22:40:26.877 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_unfound": 0, 2020-03-11T22:40:26.877 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_dirty": 100, 2020-03-11T22:40:26.877 INFO:tasks.workunit.client.0.smithi059.stdout: "num_whiteouts": 0, 2020-03-11T22:40:26.877 INFO:tasks.workunit.client.0.smithi059.stdout: "num_read": 0, 2020-03-11T22:40:26.878 INFO:tasks.workunit.client.0.smithi059.stdout: "num_read_kb": 0, 2020-03-11T22:40:26.878 INFO:tasks.workunit.client.0.smithi059.stdout: "num_write": 101, 2020-03-11T22:40:26.878 INFO:tasks.workunit.client.0.smithi059.stdout: "num_write_kb": 101, 2020-03-11T22:40:26.878 INFO:tasks.workunit.client.0.smithi059.stdout: "num_scrub_errors": 0, 2020-03-11T22:40:26.878 INFO:tasks.workunit.client.0.smithi059.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:40:26.879 INFO:tasks.workunit.client.0.smithi059.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:40:26.879 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_recovered": 0, 2020-03-11T22:40:26.879 INFO:tasks.workunit.client.0.smithi059.stdout: "num_bytes_recovered": 0, 2020-03-11T22:40:26.879 INFO:tasks.workunit.client.0.smithi059.stdout: "num_keys_recovered": 0, 2020-03-11T22:40:26.879 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_omap": 0, 2020-03-11T22:40:26.879 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:40:26.879 INFO:tasks.workunit.client.0.smithi059.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:40:26.880 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush": 0, 2020-03-11T22:40:26.880 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush_kb": 0, 2020-03-11T22:40:26.880 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict": 0, 2020-03-11T22:40:26.880 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict_kb": 0, 2020-03-11T22:40:26.880 INFO:tasks.workunit.client.0.smithi059.stdout: "num_promote": 0, 2020-03-11T22:40:26.880 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush_mode_high": 0, 2020-03-11T22:40:26.880 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush_mode_low": 0, 2020-03-11T22:40:26.880 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict_mode_some": 0, 2020-03-11T22:40:26.881 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict_mode_full": 0, 2020-03-11T22:40:26.881 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_pinned": 0, 2020-03-11T22:40:26.881 INFO:tasks.workunit.client.0.smithi059.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:40:26.881 INFO:tasks.workunit.client.0.smithi059.stdout: "num_large_omap_objects": 0, 2020-03-11T22:40:26.881 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_manifest": 0, 2020-03-11T22:40:26.881 INFO:tasks.workunit.client.0.smithi059.stdout: "num_omap_bytes": 0, 2020-03-11T22:40:26.881 INFO:tasks.workunit.client.0.smithi059.stdout: "num_omap_keys": 0, 2020-03-11T22:40:26.882 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_repaired": 0 2020-03-11T22:40:26.882 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.882 INFO:tasks.workunit.client.0.smithi059.stdout: "store_stats": { 2020-03-11T22:40:26.882 INFO:tasks.workunit.client.0.smithi059.stdout: "total": 0, 2020-03-11T22:40:26.882 INFO:tasks.workunit.client.0.smithi059.stdout: "available": 0, 2020-03-11T22:40:26.882 INFO:tasks.workunit.client.0.smithi059.stdout: "internally_reserved": 0, 2020-03-11T22:40:26.882 INFO:tasks.workunit.client.0.smithi059.stdout: "allocated": 19529728, 2020-03-11T22:40:26.883 INFO:tasks.workunit.client.0.smithi059.stdout: "data_stored": 142522, 2020-03-11T22:40:26.883 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed": 0, 2020-03-11T22:40:26.883 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_allocated": 0, 2020-03-11T22:40:26.883 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_original": 0, 2020-03-11T22:40:26.883 INFO:tasks.workunit.client.0.smithi059.stdout: "omap_allocated": 0, 2020-03-11T22:40:26.883 INFO:tasks.workunit.client.0.smithi059.stdout: "internal_metadata": 0 2020-03-11T22:40:26.883 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.883 INFO:tasks.workunit.client.0.smithi059.stdout: "log_size": 513, 2020-03-11T22:40:26.884 INFO:tasks.workunit.client.0.smithi059.stdout: "ondisk_log_size": 513, 2020-03-11T22:40:26.884 INFO:tasks.workunit.client.0.smithi059.stdout: "up": 64, 2020-03-11T22:40:26.884 INFO:tasks.workunit.client.0.smithi059.stdout: "acting": 64, 2020-03-11T22:40:26.884 INFO:tasks.workunit.client.0.smithi059.stdout: "num_store_stats": 3 2020-03-11T22:40:26.884 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:40:26.884 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:40:26.884 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_stats": [ 2020-03-11T22:40:26.885 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.885 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 0, 2020-03-11T22:40:26.885 INFO:tasks.workunit.client.0.smithi059.stdout: "seq": 158913789971 2020-03-11T22:40:26.885 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.885 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.885 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 1, 2020-03-11T22:40:26.885 INFO:tasks.workunit.client.0.smithi059.stdout: "seq": 42949672975 2020-03-11T22:40:26.886 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.886 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.886 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 2, 2020-03-11T22:40:26.886 INFO:tasks.workunit.client.0.smithi059.stdout: "seq": 167503724563 2020-03-11T22:40:26.886 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:40:26.886 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:40:26.886 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pg_by_state": [ 2020-03-11T22:40:26.886 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.887 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "active+undersized+degraded", 2020-03-11T22:40:26.887 INFO:tasks.workunit.client.0.smithi059.stdout: "num": 32 2020-03-11T22:40:26.887 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:40:26.887 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:40:26.887 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pg_by_osd": [ 2020-03-11T22:40:26.887 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.887 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 2, 2020-03-11T22:40:26.888 INFO:tasks.workunit.client.0.smithi059.stdout: "num_primary_pg": 0, 2020-03-11T22:40:26.888 INFO:tasks.workunit.client.0.smithi059.stdout: "num_acting_pg": 32, 2020-03-11T22:40:26.888 INFO:tasks.workunit.client.0.smithi059.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:40:26.888 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.888 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.888 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 1, 2020-03-11T22:40:26.888 INFO:tasks.workunit.client.0.smithi059.stdout: "num_primary_pg": 0, 2020-03-11T22:40:26.888 INFO:tasks.workunit.client.0.smithi059.stdout: "num_acting_pg": 0, 2020-03-11T22:40:26.889 INFO:tasks.workunit.client.0.smithi059.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:40:26.889 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.889 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.889 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 0, 2020-03-11T22:40:26.890 INFO:tasks.workunit.client.0.smithi059.stdout: "num_primary_pg": 32, 2020-03-11T22:40:26.890 INFO:tasks.workunit.client.0.smithi059.stdout: "num_acting_pg": 32, 2020-03-11T22:40:26.890 INFO:tasks.workunit.client.0.smithi059.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:40:26.890 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:40:26.890 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:40:26.890 INFO:tasks.workunit.client.0.smithi059.stdout: "purged_snaps": [ 2020-03-11T22:40:26.891 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:40:26.891 INFO:tasks.workunit.client.0.smithi059.stdout: "pool": 1, 2020-03-11T22:40:26.891 INFO:tasks.workunit.client.0.smithi059.stdout: "purged_snaps": {} 2020-03-11T22:40:26.891 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:40:26.891 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:40:26.891 INFO:tasks.workunit.client.0.smithi059.stdout: "servicemap": { 2020-03-11T22:40:26.891 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": 1, 2020-03-11T22:40:26.892 INFO:tasks.workunit.client.0.smithi059.stdout: "modified": "2020-03-11T22:38:08.591885+0000", 2020-03-11T22:40:26.892 INFO:tasks.workunit.client.0.smithi059.stdout: "services": {} 2020-03-11T22:40:26.892 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:40:26.892 INFO:tasks.workunit.client.0.smithi059.stdout: "paxos": { 2020-03-11T22:40:26.892 INFO:tasks.workunit.client.0.smithi059.stdout: "first_committed": 1, 2020-03-11T22:40:26.892 INFO:tasks.workunit.client.0.smithi059.stdout: "last_committed": 394, 2020-03-11T22:40:26.892 INFO:tasks.workunit.client.0.smithi059.stdout: "last_pn": 0, 2020-03-11T22:40:26.892 INFO:tasks.workunit.client.0.smithi059.stdout: "accepted_pn": 0 2020-03-11T22:40:26.893 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:40:26.893 INFO:tasks.workunit.client.0.smithi059.stdout:} 2020-03-11T22:40:26.893 INFO:tasks.workunit.client.0.smithi059.stdout:delay recovery 2020-03-11T22:40:26.893 INFO:tasks.workunit.client.0.smithi059.stderr:report 3618754875 2020-03-11T22:40:26.893 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: return 1 2020-03-11T22:40:26.894 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:746: TEST_divergent_3: echo 'delay recovery' 2020-03-11T22:40:26.894 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:747: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:40:27.034 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:40:27.034 INFO:tasks.workunit.client.0.smithi059.stdout:1.1f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.448084+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:27.034 INFO:tasks.workunit.client.0.smithi059.stdout:1.1e 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.446855+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:27.035 INFO:tasks.workunit.client.0.smithi059.stdout:1.1d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.449135+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:27.035 INFO:tasks.workunit.client.0.smithi059.stdout:1.1c 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.446971+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:27.035 INFO:tasks.workunit.client.0.smithi059.stdout:1.1b 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.446984+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:27.035 INFO:tasks.workunit.client.0.smithi059.stdout:1.1a 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.444150+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:27.035 INFO:tasks.workunit.client.0.smithi059.stdout:1.19 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.444719+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:27.035 INFO:tasks.workunit.client.0.smithi059.stdout:1.18 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.448692+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:27.036 INFO:tasks.workunit.client.0.smithi059.stdout:1.17 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.446465+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:27.036 INFO:tasks.workunit.client.0.smithi059.stdout:1.16 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.446471+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:27.036 INFO:tasks.workunit.client.0.smithi059.stdout:1.15 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.445232+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:27.036 INFO:tasks.workunit.client.0.smithi059.stdout:1.14 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.445238+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:27.036 INFO:tasks.workunit.client.0.smithi059.stdout:1.13 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.447934+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:27.036 INFO:tasks.workunit.client.0.smithi059.stdout:1.12 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.445962+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:27.036 INFO:tasks.workunit.client.0.smithi059.stdout:1.11 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.445944+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:27.037 INFO:tasks.workunit.client.0.smithi059.stdout:1.10 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.445755+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:27.037 INFO:tasks.workunit.client.0.smithi059.stdout:1.5 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.445787+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:27.037 INFO:tasks.workunit.client.0.smithi059.stdout:1.4 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.447374+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:27.037 INFO:tasks.workunit.client.0.smithi059.stdout:1.3 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.444964+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:27.037 INFO:tasks.workunit.client.0.smithi059.stdout:1.2 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.445406+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:27.037 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:30.918910+0000 31'100 44:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:27.037 INFO:tasks.workunit.client.0.smithi059.stdout:1.1 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.446245+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:27.038 INFO:tasks.workunit.client.0.smithi059.stdout:1.6 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.447557+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:27.038 INFO:tasks.workunit.client.0.smithi059.stdout:1.7 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.444141+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:27.038 INFO:tasks.workunit.client.0.smithi059.stdout:1.8 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.444152+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:27.038 INFO:tasks.workunit.client.0.smithi059.stdout:1.9 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.444388+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:27.038 INFO:tasks.workunit.client.0.smithi059.stdout:1.a 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.447345+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:27.038 INFO:tasks.workunit.client.0.smithi059.stdout:1.b 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.444733+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:27.038 INFO:tasks.workunit.client.0.smithi059.stdout:1.c 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.444867+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:27.039 INFO:tasks.workunit.client.0.smithi059.stdout:1.d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.445468+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:27.039 INFO:tasks.workunit.client.0.smithi059.stdout:1.e 3 0 3 0 0 1156 0 0 17 17 active+undersized+degraded 2020-03-11T22:39:15.445193+0000 44'101 44:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:27.039 INFO:tasks.workunit.client.0.smithi059.stdout:1.f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.446311+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:27.039 INFO:tasks.workunit.client.0.smithi059.stdout: 2020-03-11T22:40:27.039 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:40:27.040 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2020-03-11T22:40:27.045 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:748: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:40:27.046 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: get_asok_path osd.0 2020-03-11T22:40:27.046 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-11T22:40:27.046 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-11T22:40:27.046 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:40:27.046 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:40:27.046 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:40:27.047 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17931/ceph-osd.0.asok 2020-03-11T22:40:27.047 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: CEPH_ARGS= 2020-03-11T22:40:27.047 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.17931/ceph-osd.0.asok set_recovery_delay 100000 2020-03-11T22:40:27.143 INFO:tasks.workunit.client.0.smithi059.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:40:27.150 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:748: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:40:27.151 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: get_asok_path osd.2 2020-03-11T22:40:27.151 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-03-11T22:40:27.151 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-03-11T22:40:27.151 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:40:27.151 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:40:27.152 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:40:27.152 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17931/ceph-osd.2.asok 2020-03-11T22:40:27.152 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: CEPH_ARGS= 2020-03-11T22:40:27.152 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.17931/ceph-osd.2.asok set_recovery_delay 100000 2020-03-11T22:40:27.248 INFO:tasks.workunit.client.0.smithi059.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:40:27.255 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:754: TEST_divergent_3: echo 'revive divergent 1' 2020-03-11T22:40:27.255 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:755: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:40:27.255 INFO:tasks.workunit.client.0.smithi059.stdout:revive divergent 1 2020-03-11T22:40:27.599 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:40:27.599 INFO:tasks.workunit.client.0.smithi059.stdout:1.1f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.448084+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:27.599 INFO:tasks.workunit.client.0.smithi059.stdout:1.1e 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.446855+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:27.600 INFO:tasks.workunit.client.0.smithi059.stdout:1.1d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.449135+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:27.600 INFO:tasks.workunit.client.0.smithi059.stdout:1.1c 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.446971+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:27.600 INFO:tasks.workunit.client.0.smithi059.stdout:1.1b 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.446984+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:27.600 INFO:tasks.workunit.client.0.smithi059.stdout:1.1a 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.444150+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:27.600 INFO:tasks.workunit.client.0.smithi059.stdout:1.19 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.444719+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:27.600 INFO:tasks.workunit.client.0.smithi059.stdout:1.18 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.448692+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:27.600 INFO:tasks.workunit.client.0.smithi059.stdout:1.17 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.446465+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:27.601 INFO:tasks.workunit.client.0.smithi059.stdout:1.16 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.446471+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:27.601 INFO:tasks.workunit.client.0.smithi059.stdout:1.15 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.445232+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:27.601 INFO:tasks.workunit.client.0.smithi059.stdout:1.14 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.445238+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:27.601 INFO:tasks.workunit.client.0.smithi059.stdout:1.13 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.447934+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:27.601 INFO:tasks.workunit.client.0.smithi059.stdout:1.12 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.445962+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:27.601 INFO:tasks.workunit.client.0.smithi059.stdout:1.11 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.445944+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:27.601 INFO:tasks.workunit.client.0.smithi059.stdout:1.10 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.445755+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:27.602 INFO:tasks.workunit.client.0.smithi059.stdout:1.5 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.445787+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:27.602 INFO:tasks.workunit.client.0.smithi059.stdout:1.4 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.447374+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:27.602 INFO:tasks.workunit.client.0.smithi059.stdout:1.3 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.444964+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:27.602 INFO:tasks.workunit.client.0.smithi059.stdout:1.2 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.445406+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:27.602 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:30.918910+0000 31'100 44:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:27.602 INFO:tasks.workunit.client.0.smithi059.stdout:1.1 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.446245+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:27.602 INFO:tasks.workunit.client.0.smithi059.stdout:1.6 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.447557+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:27.603 INFO:tasks.workunit.client.0.smithi059.stdout:1.7 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.444141+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:27.603 INFO:tasks.workunit.client.0.smithi059.stdout:1.8 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.444152+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:27.603 INFO:tasks.workunit.client.0.smithi059.stdout:1.9 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.444388+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:27.603 INFO:tasks.workunit.client.0.smithi059.stdout:1.a 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.447345+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:27.603 INFO:tasks.workunit.client.0.smithi059.stdout:1.b 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.444733+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:27.603 INFO:tasks.workunit.client.0.smithi059.stdout:1.c 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.444867+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:27.603 INFO:tasks.workunit.client.0.smithi059.stdout:1.d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.445468+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:27.604 INFO:tasks.workunit.client.0.smithi059.stdout:1.e 3 0 3 0 0 1156 0 0 17 17 active+undersized+degraded 2020-03-11T22:39:15.445193+0000 44'101 44:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:27.604 INFO:tasks.workunit.client.0.smithi059.stdout:1.f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.446311+0000 31'100 44:128 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:27.604 INFO:tasks.workunit.client.0.smithi059.stdout: 2020-03-11T22:40:27.604 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:40:27.604 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2020-03-11T22:40:27.609 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:756: TEST_divergent_3: ceph osd set noup 2020-03-11T22:40:28.270 INFO:tasks.workunit.client.0.smithi059.stderr:noup is set 2020-03-11T22:40:28.281 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:757: TEST_divergent_3: activate_osd td/divergent-priors 1 2020-03-11T22:40:28.281 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:40:28.281 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:40:28.281 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-11T22:40:28.282 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:40:28.282 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:40:28.282 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=cb7f2e42-1934-4145-a3c3-29bdd93f8adb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:40:28.282 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:40:28.282 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:40:28.282 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:40:28.282 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:40:28.283 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:40:28.283 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:40:28.283 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:40:28.283 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:40:28.283 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:40:28.283 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:40:28.283 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:40:28.284 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:40:28.284 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:40:28.284 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:40:28.284 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17931/$cluster-$name.asok' 2020-03-11T22:40:28.284 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' 2020-03-11T22:40:28.284 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:40:28.284 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:40:28.284 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:40:28.285 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:40:28.285 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:40:28.285 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:40:28.285 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:40:28.285 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:40:28.285 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:40:28.285 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-11T22:40:28.286 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=cb7f2e42-1934-4145-a3c3-29bdd93f8adb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:40:28.286 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-03-11T22:40:28.286 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.1 2020-03-11T22:40:28.287 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-11T22:40:28.287 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:40:28.287 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:40:28.287 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:40:28.309 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:40:28.306+0000 7f6f347a3ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:28.337 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:40:28.335+0000 7f6f347a3ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:28.338 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:40:28.336+0000 7f6f347a3ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:28.678 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: return 0 2020-03-11T22:40:28.678 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:758: TEST_divergent_3: sleep 5 2020-03-11T22:40:28.854 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:40:28.852+0000 7f6f347a3ec0 -1 Falling back to public interface 2020-03-11T22:40:29.796 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:40:29.793+0000 7f6f347a3ec0 -1 osd.1 31 log_to_monitors {default=true} 2020-03-11T22:40:33.681 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:760: TEST_divergent_3: echo 'delay recovery divergent' 2020-03-11T22:40:33.682 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:761: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:40:33.682 INFO:tasks.workunit.client.0.smithi059.stdout:delay recovery divergent 2020-03-11T22:40:34.033 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:40:34.033 INFO:tasks.workunit.client.0.smithi059.stdout:1.1f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.448084+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:34.033 INFO:tasks.workunit.client.0.smithi059.stdout:1.1e 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.446855+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:34.034 INFO:tasks.workunit.client.0.smithi059.stdout:1.1d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.449135+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:34.034 INFO:tasks.workunit.client.0.smithi059.stdout:1.1c 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.446971+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:34.034 INFO:tasks.workunit.client.0.smithi059.stdout:1.1b 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.446984+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:34.034 INFO:tasks.workunit.client.0.smithi059.stdout:1.1a 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.444150+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:34.034 INFO:tasks.workunit.client.0.smithi059.stdout:1.19 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.444719+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:34.034 INFO:tasks.workunit.client.0.smithi059.stdout:1.18 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.448692+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:34.035 INFO:tasks.workunit.client.0.smithi059.stdout:1.17 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.446465+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:34.035 INFO:tasks.workunit.client.0.smithi059.stdout:1.16 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.446471+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:34.035 INFO:tasks.workunit.client.0.smithi059.stdout:1.15 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.445232+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:34.035 INFO:tasks.workunit.client.0.smithi059.stdout:1.14 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.445238+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:34.035 INFO:tasks.workunit.client.0.smithi059.stdout:1.13 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.447934+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:34.035 INFO:tasks.workunit.client.0.smithi059.stdout:1.12 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.445962+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:34.036 INFO:tasks.workunit.client.0.smithi059.stdout:1.11 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.445944+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:34.036 INFO:tasks.workunit.client.0.smithi059.stdout:1.10 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.445755+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:34.036 INFO:tasks.workunit.client.0.smithi059.stdout:1.5 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.445787+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:34.036 INFO:tasks.workunit.client.0.smithi059.stdout:1.4 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.447374+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:34.036 INFO:tasks.workunit.client.0.smithi059.stdout:1.3 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.444964+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:34.036 INFO:tasks.workunit.client.0.smithi059.stdout:1.2 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.445406+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:34.036 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:30.918910+0000 31'100 45:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:34.037 INFO:tasks.workunit.client.0.smithi059.stdout:1.1 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.446245+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:34.037 INFO:tasks.workunit.client.0.smithi059.stdout:1.6 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.447557+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:34.037 INFO:tasks.workunit.client.0.smithi059.stdout:1.7 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.444141+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:34.037 INFO:tasks.workunit.client.0.smithi059.stdout:1.8 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.444152+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:34.037 INFO:tasks.workunit.client.0.smithi059.stdout:1.9 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.444388+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:34.037 INFO:tasks.workunit.client.0.smithi059.stdout:1.a 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.447345+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:34.038 INFO:tasks.workunit.client.0.smithi059.stdout:1.b 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.444733+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:34.038 INFO:tasks.workunit.client.0.smithi059.stdout:1.c 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.444867+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:34.038 INFO:tasks.workunit.client.0.smithi059.stdout:1.d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.445468+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:34.038 INFO:tasks.workunit.client.0.smithi059.stdout:1.e 3 0 3 0 0 1156 0 0 17 17 active+undersized+degraded 2020-03-11T22:39:15.445193+0000 44'101 45:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:34.038 INFO:tasks.workunit.client.0.smithi059.stdout:1.f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:39:15.446311+0000 31'100 45:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:34.038 INFO:tasks.workunit.client.0.smithi059.stdout: 2020-03-11T22:40:34.038 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:40:34.039 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2020-03-11T22:40:34.043 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:762: TEST_divergent_3: get_asok_path osd.1 2020-03-11T22:40:34.044 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:40:34.044 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:40:34.044 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:40:34.044 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:40:34.044 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:40:34.044 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17931/ceph-osd.1.asok 2020-03-11T22:40:34.045 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:762: TEST_divergent_3: CEPH_ARGS= 2020-03-11T22:40:34.045 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:762: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.17931/ceph-osd.1.asok set_recovery_delay 100000 2020-03-11T22:40:34.141 INFO:tasks.workunit.client.0.smithi059.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:40:34.148 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:764: TEST_divergent_3: ceph osd unset noup 2020-03-11T22:40:35.027 INFO:tasks.workunit.client.0.smithi059.stderr:noup is unset 2020-03-11T22:40:35.037 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:766: TEST_divergent_3: wait_for_osd up 0 2020-03-11T22:40:35.037 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:40:35.038 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:40:35.038 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:40:35.038 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:40:35.038 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:40:35.038 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:40:35.038 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:40:35.038 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:40:35.039 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:40:35.429 INFO:tasks.workunit.client.0.smithi059.stdout:osd.0 up in weight 1 up_from 37 up_thru 43 down_at 32 last_clean_interval [5,31) [v2:127.0.0.1:6802/39814,v1:127.0.0.1:6803/39814] [v2:127.0.0.1:6804/39814,v1:127.0.0.1:6805/39814] exists,up 9e248648-07e4-4309-92d7-2283b32a88ea 2020-03-11T22:40:35.429 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:40:35.430 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:40:35.430 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:40:35.430 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:40:35.430 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:767: TEST_divergent_3: wait_for_osd up 1 2020-03-11T22:40:35.430 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:40:35.431 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:40:35.431 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:40:35.431 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:40:35.431 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:40:35.431 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:40:35.431 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:40:35.432 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:40:35.795 INFO:tasks.workunit.client.0.smithi059.stdout:osd.1 up in weight 1 up_from 48 up_thru 48 down_at 33 last_clean_interval [10,31) [v2:127.0.0.1:6818/42556,v1:127.0.0.1:6819/42556] [v2:127.0.0.1:6820/42556,v1:127.0.0.1:6821/42556] exists,up 98cb8180-5961-424c-a790-9139892a46ae 2020-03-11T22:40:35.796 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:40:35.796 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:40:35.796 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:40:35.796 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:40:35.796 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:768: TEST_divergent_3: wait_for_osd up 2 2020-03-11T22:40:35.796 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:40:35.797 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:40:35.797 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:40:35.797 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:40:35.797 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:40:35.797 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:40:35.798 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:40:35.798 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:40:36.162 INFO:tasks.workunit.client.0.smithi059.stdout:osd.2 up in weight 1 up_from 39 up_thru 0 down_at 34 last_clean_interval [15,31) [v2:127.0.0.1:6810/40008,v1:127.0.0.1:6811/40008] [v2:127.0.0.1:6812/40008,v1:127.0.0.1:6813/40008] exists,up 18b16e96-77ce-45af-9ce0-23852cb42c25 2020-03-11T22:40:36.163 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:40:36.163 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:40:36.163 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:40:36.163 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:770: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:40:36.533 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:40:36.534 INFO:tasks.workunit.client.0.smithi059.stdout:1.1f 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:40:34.764427+0000 31'106 48:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:36.534 INFO:tasks.workunit.client.0.smithi059.stdout:1.1e 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:40:34.768458+0000 31'106 48:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:36.534 INFO:tasks.workunit.client.0.smithi059.stdout:1.1d 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:40:34.764906+0000 31'106 48:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:36.534 INFO:tasks.workunit.client.0.smithi059.stdout:1.1c 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:40:34.764523+0000 31'106 48:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:36.535 INFO:tasks.workunit.client.0.smithi059.stdout:1.1b 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:40:34.764485+0000 31'106 48:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:36.535 INFO:tasks.workunit.client.0.smithi059.stdout:1.1a 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:40:34.772082+0000 31'106 48:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:36.535 INFO:tasks.workunit.client.0.smithi059.stdout:1.19 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:40:34.771586+0000 31'106 48:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:36.535 INFO:tasks.workunit.client.0.smithi059.stdout:1.18 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:40:34.765799+0000 31'106 48:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:36.536 INFO:tasks.workunit.client.0.smithi059.stdout:1.17 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:40:34.765653+0000 31'106 48:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:36.536 INFO:tasks.workunit.client.0.smithi059.stdout:1.16 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:40:34.765530+0000 31'106 48:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:36.536 INFO:tasks.workunit.client.0.smithi059.stdout:1.15 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:40:34.771297+0000 31'106 48:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:36.536 INFO:tasks.workunit.client.0.smithi059.stdout:1.14 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:40:34.771043+0000 31'106 48:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:36.536 INFO:tasks.workunit.client.0.smithi059.stdout:1.13 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:40:34.768879+0000 31'106 48:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:36.537 INFO:tasks.workunit.client.0.smithi059.stdout:1.12 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:40:34.769006+0000 31'106 48:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:36.537 INFO:tasks.workunit.client.0.smithi059.stdout:1.11 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:40:34.768644+0000 31'106 48:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:36.537 INFO:tasks.workunit.client.0.smithi059.stdout:1.10 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:40:34.770709+0000 31'106 48:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:36.537 INFO:tasks.workunit.client.0.smithi059.stdout:1.5 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:40:34.770433+0000 31'106 48:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:36.537 INFO:tasks.workunit.client.0.smithi059.stdout:1.4 4 0 0 0 0 1458 0 0 16 16 peering 2020-03-11T22:40:34.769622+0000 31'106 48:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:36.537 INFO:tasks.workunit.client.0.smithi059.stdout:1.3 4 0 0 0 0 1458 0 0 16 16 peering 2020-03-11T22:40:34.770507+0000 31'106 48:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:36.538 INFO:tasks.workunit.client.0.smithi059.stdout:1.2 4 0 0 0 0 1458 0 0 16 16 peering 2020-03-11T22:40:34.769155+0000 31'106 48:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:36.538 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:40:34.765440+0000 31'106 48:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:36.538 INFO:tasks.workunit.client.0.smithi059.stdout:1.1 4 0 0 0 0 1458 0 0 16 16 peering 2020-03-11T22:40:34.769967+0000 31'106 48:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:36.538 INFO:tasks.workunit.client.0.smithi059.stdout:1.6 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:40:34.765443+0000 31'106 48:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:36.538 INFO:tasks.workunit.client.0.smithi059.stdout:1.7 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:40:34.770759+0000 31'106 48:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:36.538 INFO:tasks.workunit.client.0.smithi059.stdout:1.8 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:40:34.771150+0000 31'106 48:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:36.539 INFO:tasks.workunit.client.0.smithi059.stdout:1.9 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:40:34.770990+0000 31'106 48:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:36.539 INFO:tasks.workunit.client.0.smithi059.stdout:1.a 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:40:34.764390+0000 31'106 48:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:36.539 INFO:tasks.workunit.client.0.smithi059.stdout:1.b 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:40:34.771767+0000 31'106 48:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:36.539 INFO:tasks.workunit.client.0.smithi059.stdout:1.c 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:40:34.770119+0000 31'106 48:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:36.539 INFO:tasks.workunit.client.0.smithi059.stdout:1.d 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:40:34.769911+0000 31'106 48:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:36.539 INFO:tasks.workunit.client.0.smithi059.stdout:1.e 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:40:34.770404+0000 31'106 48:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:36.539 INFO:tasks.workunit.client.0.smithi059.stdout:1.f 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:40:34.769629+0000 31'106 48:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:36.540 INFO:tasks.workunit.client.0.smithi059.stdout: 2020-03-11T22:40:36.540 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:40:36.541 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2020-03-11T22:40:36.543 INFO:tasks.workunit.client.0.smithi059.stdout:wait for peering 2020-03-11T22:40:36.543 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:771: TEST_divergent_3: echo 'wait for peering' 2020-03-11T22:40:36.543 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:772: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:40:36.889 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:40:36.889 INFO:tasks.workunit.client.0.smithi059.stdout:1.1f 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:40:34.764427+0000 31'106 48:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:36.889 INFO:tasks.workunit.client.0.smithi059.stdout:1.1e 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:40:34.768458+0000 31'106 48:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:36.889 INFO:tasks.workunit.client.0.smithi059.stdout:1.1d 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:40:34.764906+0000 31'106 48:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:36.890 INFO:tasks.workunit.client.0.smithi059.stdout:1.1c 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:40:34.764523+0000 31'106 48:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:36.890 INFO:tasks.workunit.client.0.smithi059.stdout:1.1b 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:40:34.764485+0000 31'106 48:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:36.890 INFO:tasks.workunit.client.0.smithi059.stdout:1.1a 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:40:34.772082+0000 31'106 48:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:36.890 INFO:tasks.workunit.client.0.smithi059.stdout:1.19 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:40:34.771586+0000 31'106 48:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:36.890 INFO:tasks.workunit.client.0.smithi059.stdout:1.18 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:40:34.765799+0000 31'106 48:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:36.890 INFO:tasks.workunit.client.0.smithi059.stdout:1.17 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:40:34.765653+0000 31'106 48:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:36.890 INFO:tasks.workunit.client.0.smithi059.stdout:1.16 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:40:34.765530+0000 31'106 48:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:36.891 INFO:tasks.workunit.client.0.smithi059.stdout:1.15 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:40:34.771297+0000 31'106 48:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:36.891 INFO:tasks.workunit.client.0.smithi059.stdout:1.14 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:40:34.771043+0000 31'106 48:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:36.891 INFO:tasks.workunit.client.0.smithi059.stdout:1.13 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:40:34.768879+0000 31'106 48:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:36.891 INFO:tasks.workunit.client.0.smithi059.stdout:1.12 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:40:34.769006+0000 31'106 48:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:36.891 INFO:tasks.workunit.client.0.smithi059.stdout:1.11 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:40:34.768644+0000 31'106 48:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:36.891 INFO:tasks.workunit.client.0.smithi059.stdout:1.10 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:40:34.770709+0000 31'106 48:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:36.891 INFO:tasks.workunit.client.0.smithi059.stdout:1.5 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:40:34.770433+0000 31'106 48:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:36.892 INFO:tasks.workunit.client.0.smithi059.stdout:1.4 4 0 0 0 0 1458 0 0 16 16 peering 2020-03-11T22:40:34.769622+0000 31'106 48:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:36.892 INFO:tasks.workunit.client.0.smithi059.stdout:1.3 4 0 0 0 0 1458 0 0 16 16 peering 2020-03-11T22:40:34.770507+0000 31'106 48:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:36.892 INFO:tasks.workunit.client.0.smithi059.stdout:1.2 4 0 0 0 0 1458 0 0 16 16 peering 2020-03-11T22:40:34.769155+0000 31'106 48:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:36.892 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:40:34.765440+0000 31'106 48:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:36.892 INFO:tasks.workunit.client.0.smithi059.stdout:1.1 4 0 0 0 0 1458 0 0 16 16 peering 2020-03-11T22:40:34.769967+0000 31'106 48:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:36.892 INFO:tasks.workunit.client.0.smithi059.stdout:1.6 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:40:34.765443+0000 31'106 48:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:36.892 INFO:tasks.workunit.client.0.smithi059.stdout:1.7 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:40:34.770759+0000 31'106 48:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:36.893 INFO:tasks.workunit.client.0.smithi059.stdout:1.8 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:40:34.771150+0000 31'106 48:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:36.893 INFO:tasks.workunit.client.0.smithi059.stdout:1.9 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:40:34.770990+0000 31'106 48:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:36.893 INFO:tasks.workunit.client.0.smithi059.stdout:1.a 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:40:34.764390+0000 31'106 48:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:36.893 INFO:tasks.workunit.client.0.smithi059.stdout:1.b 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:40:34.771767+0000 31'106 48:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:36.893 INFO:tasks.workunit.client.0.smithi059.stdout:1.c 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:40:34.770119+0000 31'106 48:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:36.893 INFO:tasks.workunit.client.0.smithi059.stdout:1.d 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:40:34.769911+0000 31'106 48:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:36.893 INFO:tasks.workunit.client.0.smithi059.stdout:1.e 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:40:34.770404+0000 31'106 48:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:36.894 INFO:tasks.workunit.client.0.smithi059.stdout:1.f 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:40:34.769629+0000 31'106 48:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:36.894 INFO:tasks.workunit.client.0.smithi059.stdout: 2020-03-11T22:40:36.894 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:40:36.894 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2020-03-11T22:40:36.898 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:773: TEST_divergent_3: rados -p test put foo /etc/fstab 2020-03-11T22:40:43.505 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:777: TEST_divergent_3: echo 'killing divergent 1' 2020-03-11T22:40:43.505 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:778: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:40:43.505 INFO:tasks.workunit.client.0.smithi059.stdout:killing divergent 1 2020-03-11T22:40:43.860 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:40:43.860 INFO:tasks.workunit.client.0.smithi059.stdout:1.1f 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:40:35.254645+0000 31'100 49:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:43.860 INFO:tasks.workunit.client.0.smithi059.stdout:1.1e 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:40:35.255827+0000 31'100 49:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:43.861 INFO:tasks.workunit.client.0.smithi059.stdout:1.1d 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:40:35.254088+0000 31'100 49:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:43.861 INFO:tasks.workunit.client.0.smithi059.stdout:1.1c 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:40:35.252140+0000 31'100 49:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:43.861 INFO:tasks.workunit.client.0.smithi059.stdout:1.1b 3 1 1 0 0 1509 0 0 10 10 active+recovery_wait+degraded+wait 2020-03-11T22:40:35.252611+0000 31'100 49:125 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:43.861 INFO:tasks.workunit.client.0.smithi059.stdout:1.1a 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:40:35.250667+0000 31'100 49:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:43.862 INFO:tasks.workunit.client.0.smithi059.stdout:1.19 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:40:35.232806+0000 31'100 49:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:43.862 INFO:tasks.workunit.client.0.smithi059.stdout:1.18 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:40:35.253838+0000 31'100 49:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:43.862 INFO:tasks.workunit.client.0.smithi059.stdout:1.17 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:40:35.251730+0000 31'100 49:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:43.862 INFO:tasks.workunit.client.0.smithi059.stdout:1.16 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:40:35.252223+0000 31'100 49:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:43.862 INFO:tasks.workunit.client.0.smithi059.stdout:1.15 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:40:35.251309+0000 31'100 49:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:43.863 INFO:tasks.workunit.client.0.smithi059.stdout:1.14 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:40:35.254971+0000 31'100 49:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:43.863 INFO:tasks.workunit.client.0.smithi059.stdout:1.13 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:40:35.252341+0000 31'100 49:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:43.863 INFO:tasks.workunit.client.0.smithi059.stdout:1.12 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:40:35.251471+0000 31'100 49:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:43.863 INFO:tasks.workunit.client.0.smithi059.stdout:1.11 3 1 1 0 0 1509 0 0 16 16 active+recovery_wait+degraded+wait 2020-03-11T22:40:35.251908+0000 31'100 49:125 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:43.864 INFO:tasks.workunit.client.0.smithi059.stdout:1.10 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:40:35.251579+0000 31'100 49:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:43.864 INFO:tasks.workunit.client.0.smithi059.stdout:1.5 3 0 0 0 0 1510 0 0 10 10 active+clean+wait 2020-03-11T22:40:35.255225+0000 31'100 49:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:43.864 INFO:tasks.workunit.client.0.smithi059.stdout:1.4 4 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:40:35.251024+0000 31'100 49:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:43.864 INFO:tasks.workunit.client.0.smithi059.stdout:1.3 4 0 0 0 0 1510 0 0 10 10 active+clean+wait 2020-03-11T22:40:35.250700+0000 31'100 49:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:43.864 INFO:tasks.workunit.client.0.smithi059.stdout:1.2 4 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:40:35.251790+0000 31'100 49:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:43.865 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:40:35.256294+0000 31'100 49:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:43.865 INFO:tasks.workunit.client.0.smithi059.stdout:1.1 4 0 0 0 0 1510 0 0 15 15 active+clean+wait 2020-03-11T22:40:35.253982+0000 31'100 49:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:43.865 INFO:tasks.workunit.client.0.smithi059.stdout:1.6 3 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:40:35.254219+0000 31'100 49:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:43.865 INFO:tasks.workunit.client.0.smithi059.stdout:1.7 3 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:40:35.232737+0000 31'100 49:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:43.866 INFO:tasks.workunit.client.0.smithi059.stdout:1.8 3 0 0 0 0 1510 0 0 10 10 active+clean+wait 2020-03-11T22:40:35.250426+0000 31'100 49:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:43.866 INFO:tasks.workunit.client.0.smithi059.stdout:1.9 3 1 1 0 0 1510 0 0 11 11 active+recovering+degraded+wait 2020-03-11T22:40:35.255422+0000 31'100 49:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:43.866 INFO:tasks.workunit.client.0.smithi059.stdout:1.a 3 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:40:35.256827+0000 31'100 49:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:43.866 INFO:tasks.workunit.client.0.smithi059.stdout:1.b 3 0 0 0 0 1510 0 0 10 10 active+clean+wait 2020-03-11T22:40:35.250986+0000 31'100 49:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:43.867 INFO:tasks.workunit.client.0.smithi059.stdout:1.c 3 0 0 0 0 1510 0 0 14 14 active+clean+wait 2020-03-11T22:40:35.251545+0000 31'100 49:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:43.867 INFO:tasks.workunit.client.0.smithi059.stdout:1.d 3 0 0 0 0 1509 0 0 12 12 active+clean+wait 2020-03-11T22:40:35.250965+0000 31'100 49:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:43.867 INFO:tasks.workunit.client.0.smithi059.stdout:1.e 3 1 1 0 0 1156 0 0 14 14 active+recovery_wait+degraded+wait 2020-03-11T22:40:35.250502+0000 44'101 49:125 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:43.867 INFO:tasks.workunit.client.0.smithi059.stdout:1.f 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:40:35.255521+0000 31'100 49:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:43.867 INFO:tasks.workunit.client.0.smithi059.stdout: 2020-03-11T22:40:43.868 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:40:43.868 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2020-03-11T22:40:43.870 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:779: TEST_divergent_3: kill_daemons td/divergent-priors KILL osd.1 2020-03-11T22:40:43.871 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:40:43.871 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:40:43.871 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:40:43.871 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:40:43.872 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:40:43.977 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:40:43.977 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:782: TEST_divergent_3: expfile=td/divergent-priors/exp.17931.out 2020-03-11T22:40:43.977 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:783: TEST_divergent_3: _objectstore_tool_nodown td/divergent-priors 1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.17931.out 2020-03-11T22:40:43.978 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/divergent-priors 2020-03-11T22:40:43.978 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-03-11T22:40:43.978 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=1 2020-03-11T22:40:43.978 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-03-11T22:40:43.978 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2020-03-11T22:40:43.978 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.17931.out 2020-03-11T22:40:44.621 INFO:tasks.workunit.client.0.smithi059.stderr:Exporting 1.0 info 1.0( v 31'100 (31'90,31'100] local-lis/les=48/49 n=3 ec=23/23 lis/c=48/48 les/c/f=49/49/0 sis=48 pruub=8.190734180s) 2020-03-11T22:40:44.635 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:023dd3d2:::existing_12:head# 2020-03-11T22:40:44.667 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:04c40a0c:::existing_93:head# 2020-03-11T22:40:44.668 INFO:tasks.workunit.client.0.smithi059.stderr:Read #1:064ff5fb:::existing_36:head# 2020-03-11T22:40:44.671 INFO:tasks.workunit.client.0.smithi059.stderr:Export successful 2020-03-11T22:40:44.671 INFO:tasks.workunit.client.0.smithi059.stdout: marking collection for removal 2020-03-11T22:40:44.671 INFO:tasks.workunit.client.0.smithi059.stdout:setting '_remove' omap key 2020-03-11T22:40:44.672 INFO:tasks.workunit.client.0.smithi059.stdout:finish_remove_pgs 1.0_head removing 1.0 2020-03-11T22:40:44.720 INFO:tasks.workunit.client.0.smithi059.stderr:Remove successful 2020-03-11T22:40:45.187 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:784: TEST_divergent_3: _objectstore_tool_nodown td/divergent-priors 1 --op import --file td/divergent-priors/exp.17931.out 2020-03-11T22:40:45.187 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/divergent-priors 2020-03-11T22:40:45.188 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-03-11T22:40:45.188 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=1 2020-03-11T22:40:45.188 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-03-11T22:40:45.188 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2020-03-11T22:40:45.188 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op import --file td/divergent-priors/exp.17931.out 2020-03-11T22:40:45.673 INFO:tasks.workunit.client.0.smithi059.stdout:get_pg_num_history pg_num_history pg_num_history(e49 pg_nums {1={23=1,43=32}} deleted_pools ) 2020-03-11T22:40:45.673 INFO:tasks.workunit.client.0.smithi059.stdout:Importing pgid 1.0 2020-03-11T22:40:45.673 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:023dd3d2:::existing_12:head# 2020-03-11T22:40:45.674 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:40:45.711 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:04c40a0c:::existing_93:head# 2020-03-11T22:40:45.711 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:40:45.719 INFO:tasks.workunit.client.0.smithi059.stdout:Write #1:064ff5fb:::existing_36:head# 2020-03-11T22:40:45.720 INFO:tasks.workunit.client.0.smithi059.stdout:snapset 0=[]:{} 2020-03-11T22:40:45.728 INFO:tasks.workunit.client.0.smithi059.stdout:write_pg epoch 49 info 1.0( v 31'100 (31'90,31'100] local-lis/les=48/49 n=3 ec=23/23 lis/c=48/48 les/c/f=49/49/0 sis=48 pruub=8.190734180s) 2020-03-11T22:40:45.736 INFO:tasks.workunit.client.0.smithi059.stdout:Import successful 2020-03-11T22:40:46.255 INFO:tasks.workunit.client.0.smithi059.stdout:reviving divergent 1 2020-03-11T22:40:46.255 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:786: TEST_divergent_3: echo 'reviving divergent 1' 2020-03-11T22:40:46.255 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:787: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:40:46.608 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:40:46.608 INFO:tasks.workunit.client.0.smithi059.stdout:1.1f 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:40:35.254645+0000 31'100 49:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:46.609 INFO:tasks.workunit.client.0.smithi059.stdout:1.1e 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:40:35.255827+0000 31'100 49:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:46.609 INFO:tasks.workunit.client.0.smithi059.stdout:1.1d 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:40:35.254088+0000 31'100 49:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:46.609 INFO:tasks.workunit.client.0.smithi059.stdout:1.1c 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:40:35.252140+0000 31'100 49:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:46.609 INFO:tasks.workunit.client.0.smithi059.stdout:1.1b 3 1 1 0 0 1509 0 0 10 10 active+recovery_wait+degraded+wait 2020-03-11T22:40:35.252611+0000 31'100 49:125 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:46.609 INFO:tasks.workunit.client.0.smithi059.stdout:1.1a 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:40:35.250667+0000 31'100 49:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:46.609 INFO:tasks.workunit.client.0.smithi059.stdout:1.19 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:40:35.232806+0000 31'100 49:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:46.609 INFO:tasks.workunit.client.0.smithi059.stdout:1.18 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:40:35.253838+0000 31'100 49:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:46.610 INFO:tasks.workunit.client.0.smithi059.stdout:1.17 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:40:35.251730+0000 31'100 49:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:46.610 INFO:tasks.workunit.client.0.smithi059.stdout:1.16 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:40:35.252223+0000 31'100 49:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:46.610 INFO:tasks.workunit.client.0.smithi059.stdout:1.15 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:40:35.251309+0000 31'100 49:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:46.610 INFO:tasks.workunit.client.0.smithi059.stdout:1.14 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:40:35.254971+0000 31'100 49:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:46.610 INFO:tasks.workunit.client.0.smithi059.stdout:1.13 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:40:35.252341+0000 31'100 49:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:46.610 INFO:tasks.workunit.client.0.smithi059.stdout:1.12 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:40:35.251471+0000 31'100 49:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:46.611 INFO:tasks.workunit.client.0.smithi059.stdout:1.11 3 1 1 0 0 1509 0 0 16 16 active+recovery_wait+degraded+wait 2020-03-11T22:40:35.251908+0000 31'100 49:125 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:46.611 INFO:tasks.workunit.client.0.smithi059.stdout:1.10 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:40:35.251579+0000 31'100 49:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:46.611 INFO:tasks.workunit.client.0.smithi059.stdout:1.5 3 0 0 0 0 1510 0 0 10 10 active+clean+wait 2020-03-11T22:40:35.255225+0000 31'100 49:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:46.611 INFO:tasks.workunit.client.0.smithi059.stdout:1.4 4 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:40:35.251024+0000 31'100 49:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:46.611 INFO:tasks.workunit.client.0.smithi059.stdout:1.3 4 0 0 0 0 1510 0 0 10 10 active+clean+wait 2020-03-11T22:40:35.250700+0000 31'100 49:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:46.611 INFO:tasks.workunit.client.0.smithi059.stdout:1.2 4 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:40:35.251790+0000 31'100 49:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:46.611 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:40:35.256294+0000 31'100 49:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:46.612 INFO:tasks.workunit.client.0.smithi059.stdout:1.1 4 0 0 0 0 1510 0 0 15 15 active+clean+wait 2020-03-11T22:40:35.253982+0000 31'100 49:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:46.612 INFO:tasks.workunit.client.0.smithi059.stdout:1.6 3 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:40:35.254219+0000 31'100 49:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:46.612 INFO:tasks.workunit.client.0.smithi059.stdout:1.7 3 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:40:35.232737+0000 31'100 49:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:46.612 INFO:tasks.workunit.client.0.smithi059.stdout:1.8 3 0 0 0 0 1510 0 0 10 10 active+clean+wait 2020-03-11T22:40:35.250426+0000 31'100 49:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:46.612 INFO:tasks.workunit.client.0.smithi059.stdout:1.9 3 1 1 0 0 1510 0 0 11 11 active+recovering+degraded+wait 2020-03-11T22:40:35.255422+0000 31'100 49:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:46.612 INFO:tasks.workunit.client.0.smithi059.stdout:1.a 3 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:40:35.256827+0000 31'100 49:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:46.612 INFO:tasks.workunit.client.0.smithi059.stdout:1.b 3 0 0 0 0 1510 0 0 10 10 active+clean+wait 2020-03-11T22:40:35.250986+0000 31'100 49:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:46.613 INFO:tasks.workunit.client.0.smithi059.stdout:1.c 3 0 0 0 0 1510 0 0 14 14 active+clean+wait 2020-03-11T22:40:35.251545+0000 31'100 49:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:46.613 INFO:tasks.workunit.client.0.smithi059.stdout:1.d 3 0 0 0 0 1509 0 0 12 12 active+clean+wait 2020-03-11T22:40:35.250965+0000 31'100 49:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:46.613 INFO:tasks.workunit.client.0.smithi059.stdout:1.e 3 1 1 0 0 1156 0 0 14 14 active+recovery_wait+degraded+wait 2020-03-11T22:40:35.250502+0000 44'101 49:125 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:46.613 INFO:tasks.workunit.client.0.smithi059.stdout:1.f 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:40:35.255521+0000 31'100 49:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:40:46.613 INFO:tasks.workunit.client.0.smithi059.stdout: 2020-03-11T22:40:46.613 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:40:46.614 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2020-03-11T22:40:46.621 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:788: TEST_divergent_3: activate_osd td/divergent-priors 1 2020-03-11T22:40:46.622 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:40:46.622 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:40:46.622 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-11T22:40:46.622 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:40:46.622 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:40:46.623 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=cb7f2e42-1934-4145-a3c3-29bdd93f8adb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:40:46.623 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:40:46.623 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:40:46.623 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:40:46.623 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:40:46.623 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:40:46.623 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:40:46.624 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:40:46.624 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:40:46.624 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:40:46.624 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:40:46.624 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:40:46.624 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:40:46.624 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:40:46.625 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:40:46.625 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17931/$cluster-$name.asok' 2020-03-11T22:40:46.625 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' 2020-03-11T22:40:46.625 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:40:46.625 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:40:46.625 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:40:46.625 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:40:46.626 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:40:46.626 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:40:46.626 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:40:46.626 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:40:46.626 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:40:46.627 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-11T22:40:46.627 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=cb7f2e42-1934-4145-a3c3-29bdd93f8adb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:40:46.627 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-03-11T22:40:46.627 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.1 2020-03-11T22:40:46.628 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-11T22:40:46.628 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:40:46.628 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:40:46.628 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:40:46.649 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:40:46.646+0000 7f1a4d59eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:46.650 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:40:46.648+0000 7f1a4d59eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:46.651 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:40:46.649+0000 7f1a4d59eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:47.000 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 1 2020-03-11T22:40:47.000 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:40:47.000 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:40:47.000 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:40:47.000 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:40:47.001 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:40:47.001 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:40:47.001 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:40:47.001 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:40:47.001 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:40:47.170 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:40:47.166+0000 7f1a4d59eec0 -1 Falling back to public interface 2020-03-11T22:40:47.356 INFO:tasks.workunit.client.0.smithi059.stdout:osd.1 up in weight 1 up_from 48 up_thru 48 down_at 33 last_clean_interval [10,31) [v2:127.0.0.1:6818/42556,v1:127.0.0.1:6819/42556] [v2:127.0.0.1:6820/42556,v1:127.0.0.1:6821/42556] exists,up 98cb8180-5961-424c-a790-9139892a46ae 2020-03-11T22:40:47.356 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:40:47.357 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:40:47.357 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:40:47.357 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:40:47.357 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:789: TEST_divergent_3: wait_for_osd up 1 2020-03-11T22:40:47.358 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:40:47.358 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:40:47.358 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:40:47.358 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:40:47.359 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:40:47.359 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:40:47.359 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:40:47.359 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:40:47.722 INFO:tasks.workunit.client.0.smithi059.stdout:osd.1 up in weight 1 up_from 48 up_thru 48 down_at 33 last_clean_interval [10,31) [v2:127.0.0.1:6818/42556,v1:127.0.0.1:6819/42556] [v2:127.0.0.1:6820/42556,v1:127.0.0.1:6821/42556] exists,up 98cb8180-5961-424c-a790-9139892a46ae 2020-03-11T22:40:47.723 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:40:47.723 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:40:47.723 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:40:47.723 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:791: TEST_divergent_3: sleep 20 2020-03-11T22:40:47.985 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:40:47.983+0000 7f1a4d59eec0 -1 osd.1 49 log_to_monitors {default=true} 2020-03-11T22:41:07.724 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:792: TEST_divergent_3: get_asok_path osd.1 2020-03-11T22:41:07.725 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:41:07.725 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:41:07.725 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:41:07.725 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:41:07.725 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:41:07.725 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17931/ceph-osd.1.asok 2020-03-11T22:41:07.726 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:792: TEST_divergent_3: CEPH_ARGS= 2020-03-11T22:41:07.726 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:792: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.17931/ceph-osd.1.asok dump_ops_in_flight 2020-03-11T22:41:07.828 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2020-03-11T22:41:07.828 INFO:tasks.workunit.client.0.smithi059.stdout: "ops": [], 2020-03-11T22:41:07.828 INFO:tasks.workunit.client.0.smithi059.stdout: "num_ops": 0 2020-03-11T22:41:07.829 INFO:tasks.workunit.client.0.smithi059.stdout:} 2020-03-11T22:41:07.836 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:794: TEST_divergent_3: echo 'allowing recovery' 2020-03-11T22:41:07.836 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:795: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:41:07.836 INFO:tasks.workunit.client.0.smithi059.stdout:allowing recovery 2020-03-11T22:41:08.182 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:41:08.183 INFO:tasks.workunit.client.0.smithi059.stdout:1.1f 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:40:59.483252+0000 31'100 58:149 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:41:08.183 INFO:tasks.workunit.client.0.smithi059.stdout:1.1e 3 0 0 0 0 1449 0 0 16 16 active+clean 2020-03-11T22:40:58.608140+0000 31'100 57:149 [0,1,2] 0 [0,1,2] 0 31'100 2020-03-11T22:40:58.608091+0000 31'100 2020-03-11T22:40:58.608091+0000 0 2020-03-11T22:41:08.183 INFO:tasks.workunit.client.0.smithi059.stdout:1.1d 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:40:55.515581+0000 31'100 59:129 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:41:08.183 INFO:tasks.workunit.client.0.smithi059.stdout:1.1c 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:40:48.851834+0000 31'100 58:144 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:41:08.183 INFO:tasks.workunit.client.0.smithi059.stdout:1.1b 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:40:59.483653+0000 31'100 58:153 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:41:08.183 INFO:tasks.workunit.client.0.smithi059.stdout:1.1a 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:40:57.386052+0000 31'100 61:150 [0,1,2] 0 [0,1,2] 0 31'100 2020-03-11T22:40:50.984091+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:41:08.183 INFO:tasks.workunit.client.0.smithi059.stdout:1.19 2 0 0 0 0 966 0 0 16 16 active+clean 2020-03-11T22:41:00.678612+0000 31'100 59:132 [2,0,1] 2 [2,0,1] 2 31'100 2020-03-11T22:41:00.678563+0000 31'100 2020-03-11T22:41:00.678563+0000 0 2020-03-11T22:41:08.184 INFO:tasks.workunit.client.0.smithi059.stdout:1.18 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:40:48.850443+0000 31'100 58:145 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:41:08.184 INFO:tasks.workunit.client.0.smithi059.stdout:1.17 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:40:59.483254+0000 31'100 58:149 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:41:08.184 INFO:tasks.workunit.client.0.smithi059.stdout:1.16 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:40:57.396142+0000 31'100 57:146 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:41:08.184 INFO:tasks.workunit.client.0.smithi059.stdout:1.15 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:40:55.514698+0000 31'100 59:129 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:41:08.184 INFO:tasks.workunit.client.0.smithi059.stdout:1.14 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:40:48.850647+0000 31'100 58:145 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:41:08.184 INFO:tasks.workunit.client.0.smithi059.stdout:1.13 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:40:59.482318+0000 31'100 58:151 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:41:08.184 INFO:tasks.workunit.client.0.smithi059.stdout:1.12 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:40:57.394915+0000 31'100 61:150 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:41:08.185 INFO:tasks.workunit.client.0.smithi059.stdout:1.11 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:40:55.513803+0000 31'100 59:129 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:41:08.185 INFO:tasks.workunit.client.0.smithi059.stdout:1.10 3 0 0 0 0 1449 0 0 10 10 active+clean 2020-03-11T22:41:00.974009+0000 31'100 58:148 [1,0,2] 1 [1,0,2] 1 31'100 2020-03-11T22:41:00.973960+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:41:08.185 INFO:tasks.workunit.client.0.smithi059.stdout:1.5 3 0 0 0 0 1510 0 0 16 16 active+clean 2020-03-11T22:40:55.514635+0000 31'100 59:129 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:41:08.185 INFO:tasks.workunit.client.0.smithi059.stdout:1.4 2 0 0 0 0 966 0 0 16 16 active+clean 2020-03-11T22:40:58.004205+0000 31'100 58:147 [1,0,2] 1 [1,0,2] 1 31'100 2020-03-11T22:40:58.004154+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:41:08.185 INFO:tasks.workunit.client.0.smithi059.stdout:1.3 4 0 0 0 0 1510 0 0 10 10 active+clean 2020-03-11T22:40:59.482816+0000 31'100 58:151 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:41:08.185 INFO:tasks.workunit.client.0.smithi059.stdout:1.2 4 0 0 0 0 1510 0 0 16 16 active+clean 2020-03-11T22:40:57.395747+0000 31'100 61:150 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:41:08.186 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:40:48.851832+0000 31'100 58:145 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:41:08.186 INFO:tasks.workunit.client.0.smithi059.stdout:1.1 4 0 0 0 0 1510 0 0 16 16 active+clean 2020-03-11T22:40:55.516090+0000 31'100 59:129 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:41:08.186 INFO:tasks.workunit.client.0.smithi059.stdout:1.6 4 0 0 0 0 1993 0 0 17 17 active+clean 2020-03-11T22:40:57.394804+0000 49'101 57:143 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:41:08.186 INFO:tasks.workunit.client.0.smithi059.stdout:1.7 2 0 0 0 0 966 0 0 16 16 active+clean 2020-03-11T22:40:59.483177+0000 31'100 58:152 [1,2,0] 1 [1,2,0] 1 31'100 2020-03-11T22:40:55.646344+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:41:08.186 INFO:tasks.workunit.client.0.smithi059.stdout:1.8 3 0 0 0 0 1449 0 0 10 10 active+clean 2020-03-11T22:40:59.026043+0000 31'100 58:148 [1,0,2] 1 [1,0,2] 1 31'100 2020-03-11T22:40:59.025990+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:41:08.186 INFO:tasks.workunit.client.0.smithi059.stdout:1.9 5 0 0 0 0 2415 0 0 16 16 active+clean 2020-03-11T22:40:59.683861+0000 31'100 59:132 [2,0,1] 2 [2,0,1] 2 31'100 2020-03-11T22:40:59.683809+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:41:08.186 INFO:tasks.workunit.client.0.smithi059.stdout:1.a 3 0 0 0 0 1510 0 0 16 16 active+clean 2020-03-11T22:40:57.396062+0000 31'100 61:150 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:41:08.187 INFO:tasks.workunit.client.0.smithi059.stdout:1.b 3 0 0 0 0 1510 0 0 10 10 active+clean 2020-03-11T22:40:59.482810+0000 31'100 58:151 [1,2,0] 1 [1,2,0] 1 31'100 2020-03-11T22:40:56.685542+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:41:08.187 INFO:tasks.workunit.client.0.smithi059.stdout:1.c 3 0 0 0 0 1449 0 0 14 14 active+clean 2020-03-11T22:40:59.992504+0000 31'100 58:148 [1,0,2] 1 [1,0,2] 1 31'100 2020-03-11T22:40:59.992458+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:41:08.187 INFO:tasks.workunit.client.0.smithi059.stdout:1.d 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:40:55.514173+0000 31'100 59:129 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:41:08.187 INFO:tasks.workunit.client.0.smithi059.stdout:1.e 3 0 0 0 0 1156 0 0 17 17 active+clean 2020-03-11T22:40:57.394654+0000 44'101 57:147 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:41:08.187 INFO:tasks.workunit.client.0.smithi059.stdout:1.f 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:40:59.469436+0000 31'100 58:149 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:41:08.187 INFO:tasks.workunit.client.0.smithi059.stdout: 2020-03-11T22:41:08.188 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:41:08.188 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2020-03-11T22:41:08.192 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:797: TEST_divergent_3: for i in $osds 2020-03-11T22:41:08.192 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:799: TEST_divergent_3: ceph tell osd.0 debug kick_recovery_wq 0 2020-03-11T22:41:08.320 INFO:tasks.workunit.client.0.smithi059.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:41:08.328 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:797: TEST_divergent_3: for i in $osds 2020-03-11T22:41:08.328 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:799: TEST_divergent_3: ceph tell osd.1 debug kick_recovery_wq 0 2020-03-11T22:41:08.456 INFO:tasks.workunit.client.0.smithi059.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:41:08.463 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:797: TEST_divergent_3: for i in $osds 2020-03-11T22:41:08.463 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:799: TEST_divergent_3: ceph tell osd.2 debug kick_recovery_wq 0 2020-03-11T22:41:08.592 INFO:tasks.workunit.client.0.smithi059.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:41:08.600 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:802: TEST_divergent_3: echo 'reading divergent objects' 2020-03-11T22:41:08.600 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:803: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:41:08.600 INFO:tasks.workunit.client.0.smithi059.stdout:reading divergent objects 2020-03-11T22:41:08.942 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:41:08.942 INFO:tasks.workunit.client.0.smithi059.stdout:1.1f 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:40:59.483252+0000 31'100 62:153 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:41:08.942 INFO:tasks.workunit.client.0.smithi059.stdout:1.1e 3 0 0 0 0 1449 0 0 10 10 active+clean 2020-03-11T22:41:05.490124+0000 31'100 62:153 [1,0,2] 1 [1,0,2] 1 31'100 2020-03-11T22:40:58.608091+0000 31'100 2020-03-11T22:40:58.608091+0000 0 2020-03-11T22:41:08.942 INFO:tasks.workunit.client.0.smithi059.stdout:1.1d 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:40:55.515581+0000 31'100 62:132 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:41:08.942 INFO:tasks.workunit.client.0.smithi059.stdout:1.1c 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:40:48.851834+0000 31'100 62:148 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:41:08.942 INFO:tasks.workunit.client.0.smithi059.stdout:1.1b 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:40:59.483653+0000 31'100 62:157 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:41:08.943 INFO:tasks.workunit.client.0.smithi059.stdout:1.1a 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:40:57.386052+0000 31'100 61:150 [0,1,2] 0 [0,1,2] 0 31'100 2020-03-11T22:40:50.984091+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:41:08.943 INFO:tasks.workunit.client.0.smithi059.stdout:1.19 2 0 0 0 0 966 0 0 16 16 active+clean 2020-03-11T22:41:00.678612+0000 31'100 62:135 [2,0,1] 2 [2,0,1] 2 31'100 2020-03-11T22:41:00.678563+0000 31'100 2020-03-11T22:41:00.678563+0000 0 2020-03-11T22:41:08.943 INFO:tasks.workunit.client.0.smithi059.stdout:1.18 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:40:48.850443+0000 31'100 62:149 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:41:08.943 INFO:tasks.workunit.client.0.smithi059.stdout:1.17 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:40:59.483254+0000 31'100 62:153 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:41:08.943 INFO:tasks.workunit.client.0.smithi059.stdout:1.16 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:41:05.573744+0000 31'100 62:151 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:41:08.943 INFO:tasks.workunit.client.0.smithi059.stdout:1.15 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:40:55.514698+0000 31'100 62:132 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:41:08.943 INFO:tasks.workunit.client.0.smithi059.stdout:1.14 3 0 0 0 0 1449 0 0 10 10 active+clean 2020-03-11T22:41:07.947779+0000 31'100 63:152 [1,0,2] 1 [1,0,2] 1 31'100 2020-03-11T22:41:07.947739+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:41:08.944 INFO:tasks.workunit.client.0.smithi059.stdout:1.13 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:40:59.482318+0000 31'100 62:155 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:41:08.944 INFO:tasks.workunit.client.0.smithi059.stdout:1.12 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:40:57.394915+0000 31'100 61:150 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:41:08.944 INFO:tasks.workunit.client.0.smithi059.stdout:1.11 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:40:55.513803+0000 31'100 62:132 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:41:08.944 INFO:tasks.workunit.client.0.smithi059.stdout:1.10 3 0 0 0 0 1449 0 0 10 10 active+clean 2020-03-11T22:41:00.974009+0000 31'100 62:152 [1,0,2] 1 [1,0,2] 1 31'100 2020-03-11T22:41:00.973960+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:41:08.944 INFO:tasks.workunit.client.0.smithi059.stdout:1.5 3 0 0 0 0 1510 0 0 16 16 active+clean 2020-03-11T22:40:55.514635+0000 31'100 62:132 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:41:08.944 INFO:tasks.workunit.client.0.smithi059.stdout:1.4 2 0 0 0 0 966 0 0 16 16 active+clean 2020-03-11T22:40:58.004205+0000 31'100 62:151 [1,0,2] 1 [1,0,2] 1 31'100 2020-03-11T22:40:58.004154+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:41:08.944 INFO:tasks.workunit.client.0.smithi059.stdout:1.3 4 0 0 0 0 1510 0 0 10 10 active+clean 2020-03-11T22:40:59.482816+0000 31'100 62:155 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:41:08.945 INFO:tasks.workunit.client.0.smithi059.stdout:1.2 4 0 0 0 0 1510 0 0 16 16 active+clean 2020-03-11T22:40:57.395747+0000 31'100 61:150 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:41:08.945 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:40:48.851832+0000 31'100 62:149 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:41:08.945 INFO:tasks.workunit.client.0.smithi059.stdout:1.1 4 0 0 0 0 1510 0 0 16 16 active+clean 2020-03-11T22:40:55.516090+0000 31'100 62:132 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:41:08.945 INFO:tasks.workunit.client.0.smithi059.stdout:1.6 4 0 0 0 0 1993 0 0 17 17 active+clean 2020-03-11T22:41:05.573742+0000 49'101 62:153 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:41:08.945 INFO:tasks.workunit.client.0.smithi059.stdout:1.7 2 0 0 0 0 966 0 0 16 16 active+clean 2020-03-11T22:40:59.483177+0000 31'100 62:156 [1,2,0] 1 [1,2,0] 1 31'100 2020-03-11T22:40:55.646344+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:41:08.945 INFO:tasks.workunit.client.0.smithi059.stdout:1.8 3 0 0 0 0 1449 0 0 10 10 active+clean 2020-03-11T22:40:59.026043+0000 31'100 62:152 [1,0,2] 1 [1,0,2] 1 31'100 2020-03-11T22:40:59.025990+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:41:08.946 INFO:tasks.workunit.client.0.smithi059.stdout:1.9 5 0 0 0 0 2415 0 0 16 16 active+clean 2020-03-11T22:40:59.683861+0000 31'100 62:135 [2,0,1] 2 [2,0,1] 2 31'100 2020-03-11T22:40:59.683809+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:41:08.946 INFO:tasks.workunit.client.0.smithi059.stdout:1.a 3 0 0 0 0 1510 0 0 16 16 active+clean 2020-03-11T22:40:57.396062+0000 31'100 61:150 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:41:08.946 INFO:tasks.workunit.client.0.smithi059.stdout:1.b 3 0 0 0 0 1510 0 0 10 10 active+clean 2020-03-11T22:40:59.482810+0000 31'100 62:155 [1,2,0] 1 [1,2,0] 1 31'100 2020-03-11T22:40:56.685542+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:41:08.946 INFO:tasks.workunit.client.0.smithi059.stdout:1.c 3 0 0 0 0 1449 0 0 14 14 active+clean 2020-03-11T22:40:59.992504+0000 31'100 62:152 [1,0,2] 1 [1,0,2] 1 31'100 2020-03-11T22:40:59.992458+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:41:08.946 INFO:tasks.workunit.client.0.smithi059.stdout:1.d 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:40:55.514173+0000 31'100 62:132 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:41:08.946 INFO:tasks.workunit.client.0.smithi059.stdout:1.e 4 0 0 0 0 1579 0 0 14 14 active+clean 2020-03-11T22:41:05.490147+0000 44'101 62:155 [1,0,2] 1 [1,0,2] 1 44'101 2020-03-11T22:41:02.500633+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:41:08.946 INFO:tasks.workunit.client.0.smithi059.stdout:1.f 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:40:59.469436+0000 31'100 62:153 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:38:30.877680+0000 0'0 2020-03-11T22:38:30.877680+0000 0 2020-03-11T22:41:08.946 INFO:tasks.workunit.client.0.smithi059.stdout: 2020-03-11T22:41:08.947 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:41:08.947 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2020-03-11T22:41:08.952 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: expr 2 + 2 2020-03-11T22:41:08.953 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: seq 1 4 2020-03-11T22:41:08.953 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:41:08.954 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: rados -p test get existing_1 td/divergent-priors/existing 2020-03-11T22:41:09.045 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:41:09.045 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: rados -p test get existing_2 td/divergent-priors/existing 2020-03-11T22:41:09.092 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:41:09.092 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: rados -p test get existing_3 td/divergent-priors/existing 2020-03-11T22:41:09.130 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:41:09.130 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: rados -p test get existing_4 td/divergent-priors/existing 2020-03-11T22:41:09.197 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:808: TEST_divergent_3: seq 1 2 2020-03-11T22:41:09.197 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:808: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2020-03-11T22:41:09.197 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:810: TEST_divergent_3: rados -p test get newobject_1 td/divergent-priors/existing 2020-03-11T22:41:09.226 INFO:tasks.workunit.client.0.smithi059.stderr:error getting test/newobject_1: (2) No such file or directory 2020-03-11T22:41:09.227 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:808: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2020-03-11T22:41:09.227 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:810: TEST_divergent_3: rados -p test get newobject_2 td/divergent-priors/existing 2020-03-11T22:41:09.281 INFO:tasks.workunit.client.0.smithi059.stderr:error getting test/newobject_2: (2) No such file or directory 2020-03-11T22:41:09.282 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:812: TEST_divergent_3: rm -f td/divergent-priors/existing 2020-03-11T22:41:09.284 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2020-03-11T22:41:09.285 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: grep _merge_object_divergent_entries td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:41:09.292 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:34.868+0000 7f6f1095a700 20 _merge_object_divergent_entries: merging hoid 1:97db2a0f:::existing_2:head entries: 31'102 (31'2) modify 1:97db2a0f:::existing_2:head by client.4531.0:1 2020-03-11T22:38:55.803695+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:41:09.292 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:34.868+0000 7f6f1095a700 20 _merge_object_divergent_entries: keeping 31'102 (31'2) modify 1:97db2a0f:::existing_2:head by client.4531.0:1 2020-03-11T22:38:55.803695+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:41:09.292 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:34.868+0000 7f6f1095a700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:41:09.292 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:34.868+0000 7f6f1095a700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version: 31'2 first_divergent_update: 31'102 last_divergent_update: 31'102 2020-03-11T22:41:09.293 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:34.868+0000 7f6f1095a700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head has no more recent entries in log 2020-03-11T22:41:09.293 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:34.868+0000 7f6f1095a700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head must be rolled back or recovered, attempting to rollback 2020-03-11T22:41:09.293 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:34.868+0000 7f6f1095a700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head olog_can_rollback_to: 31'106 2020-03-11T22:41:09.293 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:34.868+0000 7f6f1095a700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot rollback 31'102 (31'2) modify 1:97db2a0f:::existing_2:head by client.4531.0:1 2020-03-11T22:38:55.803695+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:41:09.293 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:34.868+0000 7f6f1095a700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot roll back, removing and adding to missing 2020-03-11T22:41:09.293 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:34.868+0000 7f6f1095a700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version 31'2 <= info.log_tail 31'89 2020-03-11T22:41:09.294 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:34.870+0000 7f6f1095a700 20 _merge_object_divergent_entries: merging hoid 1:70b3bf12:::existing_4:head entries: 31'106 (31'4) delete 1:70b3bf12:::existing_4:head by client.4543.0:1 2020-03-11T22:38:55.820463+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:41:09.294 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:34.870+0000 7f6f1095a700 20 _merge_object_divergent_entries: keeping 31'106 (31'4) delete 1:70b3bf12:::existing_4:head by client.4543.0:1 2020-03-11T22:38:55.820463+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:41:09.294 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:34.870+0000 7f6f1095a700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:41:09.294 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:34.870+0000 7f6f1095a700 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head prior_version: 31'4 first_divergent_update: 31'106 last_divergent_update: 31'106 2020-03-11T22:41:09.294 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:34.870+0000 7f6f1095a700 10 _merge_object_divergent_entries: more recent entry found: 44'101 (31'4) modify 1:70b3bf12:::existing_4:head by client.4654.0:1 2020-03-11T22:39:50.867327+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0, already merged 2020-03-11T22:41:09.294 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:34.870+0000 7f6f1095a700 20 _merge_object_divergent_entries: merging hoid 1:75ac5091:::newobject_2:head entries: 31'103 (0'0) modify 1:75ac5091:::newobject_2:head by client.4535.0:1 2020-03-11T22:38:55.813057+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:41:09.294 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:34.870+0000 7f6f1095a700 20 _merge_object_divergent_entries: keeping 31'103 (0'0) modify 1:75ac5091:::newobject_2:head by client.4535.0:1 2020-03-11T22:38:55.813057+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:41:09.295 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:34.870+0000 7f6f1095a700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:41:09.295 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:34.870+0000 7f6f1095a700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version: 0'0 first_divergent_update: 31'103 last_divergent_update: 31'103 2020-03-11T22:41:09.295 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:34.870+0000 7f6f1095a700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head has no more recent entries in log 2020-03-11T22:41:09.295 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:34.870+0000 7f6f1095a700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version or op type indicates creation, deleting 2020-03-11T22:41:09.295 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:34.871+0000 7f6f1195c700 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 31'105 (31'3) delete 1:8881e6d8:::existing_3:head by client.4542.0:1 2020-03-11T22:38:55.820226+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:41:09.295 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:34.871+0000 7f6f1195c700 20 _merge_object_divergent_entries: keeping 31'105 (31'3) delete 1:8881e6d8:::existing_3:head by client.4542.0:1 2020-03-11T22:38:55.820226+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:41:09.295 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:34.871+0000 7f6f1195c700 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2020-03-11T22:41:09.296 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:34.871+0000 7f6f1195c700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 31'3 first_divergent_update: 31'105 last_divergent_update: 31'105 2020-03-11T22:41:09.296 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:34.871+0000 7f6f1195c700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2020-03-11T22:41:09.296 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:34.871+0000 7f6f1195c700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2020-03-11T22:41:09.296 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:34.871+0000 7f6f1195c700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 31'106 2020-03-11T22:41:09.296 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:34.871+0000 7f6f1195c700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 31'105 (31'3) delete 1:8881e6d8:::existing_3:head by client.4542.0:1 2020-03-11T22:38:55.820226+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:41:09.296 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:34.871+0000 7f6f1195c700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2020-03-11T22:41:09.296 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:34.871+0000 7f6f1195c700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version 31'3 <= info.log_tail 31'84 2020-03-11T22:41:09.297 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:34.871+0000 7f6f1195c700 20 _merge_object_divergent_entries: merging hoid 1:d858c035:::existing_1:head entries: 31'104 (31'1) modify 1:d858c035:::existing_1:head by client.4539.0:1 2020-03-11T22:38:55.817993+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:41:09.297 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:34.871+0000 7f6f1195c700 20 _merge_object_divergent_entries: keeping 31'104 (31'1) modify 1:d858c035:::existing_1:head by client.4539.0:1 2020-03-11T22:38:55.817993+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:41:09.297 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:34.871+0000 7f6f1195c700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:41:09.299 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:34.871+0000 7f6f1195c700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version: 31'1 first_divergent_update: 31'104 last_divergent_update: 31'104 2020-03-11T22:41:09.299 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:34.871+0000 7f6f1195c700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head has no more recent entries in log 2020-03-11T22:41:09.299 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:34.871+0000 7f6f1195c700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head must be rolled back or recovered, attempting to rollback 2020-03-11T22:41:09.300 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:34.871+0000 7f6f1195c700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head olog_can_rollback_to: 31'106 2020-03-11T22:41:09.300 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:34.871+0000 7f6f1195c700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot rollback 31'104 (31'1) modify 1:d858c035:::existing_1:head by client.4539.0:1 2020-03-11T22:38:55.817993+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:41:09.300 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:34.871+0000 7f6f1195c700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot roll back, removing and adding to missing 2020-03-11T22:41:09.300 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:34.871+0000 7f6f1195c700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version 31'1 <= info.log_tail 31'90 2020-03-11T22:41:09.300 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:34.871+0000 7f6f1195c700 20 _merge_object_divergent_entries: merging hoid 1:da59f6e7:::newobject_1:head entries: 31'101 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4529.0:1 2020-03-11T22:38:55.800396+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:41:09.301 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:34.871+0000 7f6f1195c700 20 _merge_object_divergent_entries: keeping 31'101 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4529.0:1 2020-03-11T22:38:55.800396+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:41:09.301 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:34.871+0000 7f6f1195c700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:41:09.301 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:34.871+0000 7f6f1195c700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version: 0'0 first_divergent_update: 31'101 last_divergent_update: 31'101 2020-03-11T22:41:09.301 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:34.871+0000 7f6f1195c700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head has no more recent entries in log 2020-03-11T22:41:09.301 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:40:34.871+0000 7f6f1195c700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version or op type indicates creation, deleting 2020-03-11T22:41:09.302 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:816: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2020-03-11T22:41:09.302 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:816: TEST_divergent_3: grep -q '_merge_object_divergent_entries: more recent entry found:' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:41:09.308 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:822: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2020-03-11T22:41:09.309 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:822: TEST_divergent_3: grep -q '_merge_object_divergent_entries.*prior_version or op type indicates creation' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:41:09.317 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:827: TEST_divergent_3: echo success 2020-03-11T22:41:09.317 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:829: TEST_divergent_3: rm td/divergent-priors/td/divergent-priors/exp.17931.out 2020-03-11T22:41:09.317 INFO:tasks.workunit.client.0.smithi059.stdout:success 2020-03-11T22:41:09.318 INFO:tasks.workunit.client.0.smithi059.stderr:rm: cannot remove 'td/divergent-priors/td/divergent-priors/exp.17931.out': No such file or directory 2020-03-11T22:41:09.319 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:831: TEST_divergent_3: delete_pool test 2020-03-11T22:41:09.319 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test 2020-03-11T22:41:09.320 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2020-03-11T22:41:09.888 INFO:tasks.workunit.client.0.smithi059.stderr:pool 'test' does not exist 2020-03-11T22:41:09.896 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:832: TEST_divergent_3: kill_daemons td/divergent-priors 2020-03-11T22:41:09.897 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:41:09.897 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:41:09.897 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:41:09.897 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:41:09.897 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:41:10.005 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:41:10.005 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2020-03-11T22:41:10.006 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-11T22:41:10.006 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:41:10.006 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-11T22:41:10.006 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:41:10.006 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:41:10.006 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:41:10.007 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:41:10.007 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:41:10.010 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:41:10.010 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:41:10.011 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:41:10.011 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:41:10.012 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:41:10.012 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:41:10.013 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:41:10.014 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:41:10.014 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:41:10.015 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:41:10.015 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:41:10.015 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:41:10.016 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:41:10.016 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-11T22:41:10.033 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:41:10.034 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:41:10.034 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:41:10.034 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17931 2020-03-11T22:41:10.035 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:41:10.036 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:41:10.036 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2020-03-11T22:41:10.036 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2020-03-11T22:41:10.036 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2020-03-11T22:41:10.036 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2020-03-11T22:41:10.036 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-11T22:41:10.037 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:41:10.037 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-11T22:41:10.037 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:41:10.037 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:41:10.037 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:41:10.037 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:41:10.037 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:41:10.038 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:41:10.038 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:41:10.038 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:41:10.039 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:41:10.039 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:41:10.039 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:41:10.040 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:41:10.041 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:41:10.041 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:41:10.042 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:41:10.042 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:41:10.042 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:41:10.043 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:41:10.043 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-11T22:41:10.044 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:41:10.044 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:41:10.045 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:41:10.045 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17931 2020-03-11T22:41:10.045 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:41:10.046 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:41:10.046 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2020-03-11T22:41:10.046 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-11T22:41:10.047 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:41:10.047 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:41:10.047 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.17931 2020-03-11T22:41:10.048 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-11T22:41:10.049 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-03-11T22:41:10.049 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-11T22:41:10.049 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2020-03-11T22:41:10.049 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_ec td/divergent-priors 2020-03-11T22:41:10.050 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:228: TEST_divergent_ec: local dir=td/divergent-priors 2020-03-11T22:41:10.050 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:231: TEST_divergent_ec: local dummyfile=/etc/fstab 2020-03-11T22:41:10.050 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:232: TEST_divergent_ec: local dummyfile2=/etc/resolv.conf 2020-03-11T22:41:10.050 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:234: TEST_divergent_ec: local num_osds=3 2020-03-11T22:41:10.050 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: expr 3 - 1 2020-03-11T22:41:10.051 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: seq 0 2 2020-03-11T22:41:10.052 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: local 'osds=0 2020-03-11T22:41:10.052 INFO:tasks.workunit.client.0.smithi059.stderr:1 2020-03-11T22:41:10.052 INFO:tasks.workunit.client.0.smithi059.stderr:2' 2020-03-11T22:41:10.052 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:236: TEST_divergent_ec: run_mon td/divergent-priors a 2020-03-11T22:41:10.052 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2020-03-11T22:41:10.052 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-11T22:41:10.052 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-11T22:41:10.053 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-11T22:41:10.053 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2020-03-11T22:41:10.053 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2020-03-11T22:41:10.370 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-11T22:41:10.370 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:41:10.370 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:41:10.370 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:41:10.370 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:41:10.370 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:41:10.371 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17931/$cluster-$name.asok' 2020-03-11T22:41:10.371 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-11T22:41:10.562 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-11T22:41:10.565 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-11T22:41:10.565 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:41:10.566 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:41:10.566 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-11T22:41:10.566 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-11T22:41:10.566 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:41:10.566 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:41:10.566 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:41:10.567 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:41:10.567 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:41:10.567 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:41:10.567 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17931/ceph-mon.a.asok 2020-03-11T22:41:10.567 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:41:10.567 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.17931/ceph-mon.a.asok config get fsid 2020-03-11T22:41:10.659 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-11T22:41:10.659 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:41:10.659 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:41:10.660 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-11T22:41:10.660 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-11T22:41:10.660 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:41:10.660 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:41:10.660 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:41:10.661 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:41:10.661 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:41:10.661 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:41:10.661 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17931/ceph-mon.a.asok 2020-03-11T22:41:10.662 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:41:10.662 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.17931/ceph-mon.a.asok config get mon_host 2020-03-11T22:41:10.753 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:237: TEST_divergent_ec: run_mgr td/divergent-priors x 2020-03-11T22:41:10.754 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/divergent-priors 2020-03-11T22:41:10.754 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-11T22:41:10.754 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-11T22:41:10.754 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-11T22:41:10.754 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/divergent-priors/x 2020-03-11T22:41:10.755 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-11T22:41:11.654 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-11T22:41:11.654 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:41:11.654 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:41:11.654 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:41:11.655 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:41:11.655 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:41:11.655 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17931/$cluster-$name.asok' 2020-03-11T22:41:11.655 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:41:11.656 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:41:11.684 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2020-03-11T22:41:11.684 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 0 2020-03-11T22:41:11.684 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:41:11.685 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:41:11.685 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-11T22:41:11.685 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:41:11.685 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/0 2020-03-11T22:41:11.685 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=cb7f2e42-1934-4145-a3c3-29bdd93f8adb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:41:11.685 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:41:11.686 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:41:11.686 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:41:11.686 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-11T22:41:11.686 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-11T22:41:11.686 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:41:11.686 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:41:11.686 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:41:11.687 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:41:11.687 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:41:11.687 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:41:11.687 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:41:11.687 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:41:11.687 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:41:11.687 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17931/$cluster-$name.asok' 2020-03-11T22:41:11.688 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' 2020-03-11T22:41:11.688 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:41:11.688 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:41:11.688 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:41:11.688 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:41:11.688 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:41:11.688 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:41:11.688 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:41:11.689 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:41:11.689 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:41:11.689 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:41:11.689 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/0 2020-03-11T22:41:11.689 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:41:11.690 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=af7f49a6-1c6c-446f-802f-d448ee247966 2020-03-11T22:41:11.690 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 af7f49a6-1c6c-446f-802f-d448ee247966' 2020-03-11T22:41:11.690 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:41:11.690 INFO:tasks.workunit.client.0.smithi059.stdout:add osd0 af7f49a6-1c6c-446f-802f-d448ee247966 2020-03-11T22:41:11.718 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQAHaWle98efKhAABYrsLFjbfMbIurgk+Ff7RA== 2020-03-11T22:41:11.718 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQAHaWle98efKhAABYrsLFjbfMbIurgk+Ff7RA=="}' 2020-03-11T22:41:11.718 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new af7f49a6-1c6c-446f-802f-d448ee247966 -i td/divergent-priors/0/new.json 2020-03-11T22:41:12.040 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:41:12.049 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/0/new.json 2020-03-11T22:41:12.050 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=cb7f2e42-1934-4145-a3c3-29bdd93f8adb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAHaWle98efKhAABYrsLFjbfMbIurgk+Ff7RA== --osd-uuid af7f49a6-1c6c-446f-802f-d448ee247966 2020-03-11T22:41:12.080 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:41:12.076+0000 7fdca15b8ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:41:12.082 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:41:12.078+0000 7fdca15b8ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:41:12.082 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:41:12.079+0000 7fdca15b8ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:41:12.083 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:41:12.079+0000 7fdca15b8ec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:41:12.083 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:41:12.079+0000 7fdca15b8ec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:41:12.083 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:41:12.079+0000 7fdca15b8ec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:41:12.119 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:41:12.116+0000 7fdca15b8ec0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2020-03-11T22:41:15.150 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/0/keyring 2020-03-11T22:41:15.150 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:41:15.151 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-11T22:41:15.151 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:41:15.152 INFO:tasks.workunit.client.0.smithi059.stdout:adding osd0 key to auth repository 2020-03-11T22:41:15.630 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.0 2020-03-11T22:41:15.630 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-11T22:41:15.631 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=cb7f2e42-1934-4145-a3c3-29bdd93f8adb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:41:15.631 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:41:15.631 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:41:15.631 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:41:15.658 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:41:15.654+0000 7f15a9a91ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:41:15.662 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:41:15.658+0000 7f15a9a91ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:41:15.663 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:41:15.659+0000 7f15a9a91ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:41:15.996 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-11T22:41:15.996 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:41:15.996 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:41:15.997 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:41:15.997 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:41:15.997 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:41:15.997 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:41:15.998 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:41:15.998 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:41:15.998 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:41:16.182 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:41:16.178+0000 7f15a9a91ec0 -1 Falling back to public interface 2020-03-11T22:41:16.384 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:41:16.910 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:41:16.907+0000 7f15a9a91ec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-11T22:41:17.386 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:41:17.386 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:41:17.386 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:41:17.386 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:41:17.386 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:41:17.387 INFO:tasks.workunit.client.0.smithi059.stdout:1 2020-03-11T22:41:17.773 INFO:tasks.workunit.client.0.smithi059.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/44058,v1:127.0.0.1:6803/44058] [v2:127.0.0.1:6804/44058,v1:127.0.0.1:6805/44058] exists,up af7f49a6-1c6c-446f-802f-d448ee247966 2020-03-11T22:41:17.773 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:41:17.773 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:41:17.773 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:41:17.774 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2020-03-11T22:41:17.774 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 1 2020-03-11T22:41:17.774 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:41:17.774 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:41:17.774 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-11T22:41:17.774 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:41:17.775 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:41:17.775 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=cb7f2e42-1934-4145-a3c3-29bdd93f8adb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:41:17.775 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:41:17.775 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:41:17.775 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:41:17.775 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:41:17.775 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:41:17.776 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:41:17.776 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:41:17.776 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:41:17.776 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:41:17.776 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:41:17.776 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:41:17.776 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:41:17.776 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:41:17.777 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:41:17.777 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17931/$cluster-$name.asok' 2020-03-11T22:41:17.777 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' 2020-03-11T22:41:17.777 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:41:17.777 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:41:17.777 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:41:17.777 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:41:17.778 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:41:17.778 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:41:17.778 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:41:17.779 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:41:17.779 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:41:17.779 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:41:17.779 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:41:17.779 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:41:17.780 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=01a46a03-2514-4118-9d87-d9feba7ccc07 2020-03-11T22:41:17.781 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 01a46a03-2514-4118-9d87-d9feba7ccc07' 2020-03-11T22:41:17.782 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:41:17.782 INFO:tasks.workunit.client.0.smithi059.stdout:add osd1 01a46a03-2514-4118-9d87-d9feba7ccc07 2020-03-11T22:41:17.793 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQANaWleMuYgLxAA4giswps19jV3gw2y8ggGmA== 2020-03-11T22:41:17.793 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQANaWleMuYgLxAA4giswps19jV3gw2y8ggGmA=="}' 2020-03-11T22:41:17.794 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 01a46a03-2514-4118-9d87-d9feba7ccc07 -i td/divergent-priors/1/new.json 2020-03-11T22:41:18.290 INFO:tasks.workunit.client.0.smithi059.stdout:1 2020-03-11T22:41:18.300 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/1/new.json 2020-03-11T22:41:18.301 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=cb7f2e42-1934-4145-a3c3-29bdd93f8adb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQANaWleMuYgLxAA4giswps19jV3gw2y8ggGmA== --osd-uuid 01a46a03-2514-4118-9d87-d9feba7ccc07 2020-03-11T22:41:18.331 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:41:18.327+0000 7fcbb7330ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:41:18.351 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:41:18.347+0000 7fcbb7330ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:41:18.352 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:41:18.349+0000 7fcbb7330ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:41:18.353 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:41:18.349+0000 7fcbb7330ec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:41:18.353 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:41:18.349+0000 7fcbb7330ec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:41:18.353 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:41:18.349+0000 7fcbb7330ec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:41:18.459 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:41:18.456+0000 7fcbb7330ec0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2020-03-11T22:41:21.585 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/1/keyring 2020-03-11T22:41:21.585 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:41:21.586 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-03-11T22:41:21.586 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:41:21.586 INFO:tasks.workunit.client.0.smithi059.stdout:adding osd1 key to auth repository 2020-03-11T22:41:22.077 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-03-11T22:41:22.077 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:41:22.077 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:41:22.078 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:41:22.078 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=cb7f2e42-1934-4145-a3c3-29bdd93f8adb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:41:22.078 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.1 2020-03-11T22:41:22.105 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:41:22.101+0000 7f7dde90fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:41:22.107 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:41:22.103+0000 7f7dde90fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:41:22.108 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:41:22.104+0000 7f7dde90fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:41:22.444 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-03-11T22:41:22.445 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:41:22.445 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:41:22.445 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:41:22.445 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:41:22.445 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:41:22.445 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:41:22.446 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:41:22.446 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:41:22.446 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:41:22.623 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:41:22.619+0000 7f7dde90fec0 -1 Falling back to public interface 2020-03-11T22:41:22.806 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:41:23.442 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:41:23.438+0000 7f7dde90fec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-11T22:41:23.808 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:41:23.808 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:41:23.808 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:41:23.808 INFO:tasks.workunit.client.0.smithi059.stdout:1 2020-03-11T22:41:23.809 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:41:23.809 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:41:24.189 INFO:tasks.workunit.client.0.smithi059.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/44413,v1:127.0.0.1:6811/44413] [v2:127.0.0.1:6812/44413,v1:127.0.0.1:6813/44413] exists,up 01a46a03-2514-4118-9d87-d9feba7ccc07 2020-03-11T22:41:24.189 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:41:24.189 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:41:24.190 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:41:24.190 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2020-03-11T22:41:24.190 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 2 2020-03-11T22:41:24.190 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:41:24.190 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:41:24.190 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-03-11T22:41:24.190 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:41:24.191 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/2 2020-03-11T22:41:24.191 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=cb7f2e42-1934-4145-a3c3-29bdd93f8adb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:41:24.191 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:41:24.191 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:41:24.191 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:41:24.191 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-11T22:41:24.191 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-11T22:41:24.192 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:41:24.192 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:41:24.192 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:41:24.192 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:41:24.192 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:41:24.192 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:41:24.192 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:41:24.193 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:41:24.194 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:41:24.194 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17931/$cluster-$name.asok' 2020-03-11T22:41:24.194 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' 2020-03-11T22:41:24.194 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:41:24.194 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:41:24.194 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:41:24.195 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:41:24.195 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:41:24.195 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:41:24.195 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:41:24.195 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:41:24.195 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:41:24.196 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:41:24.196 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/2 2020-03-11T22:41:24.196 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:41:24.197 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=d2482f9a-6ef9-4322-9e82-b5d8b8810499 2020-03-11T22:41:24.197 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 d2482f9a-6ef9-4322-9e82-b5d8b8810499' 2020-03-11T22:41:24.197 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:41:24.198 INFO:tasks.workunit.client.0.smithi059.stdout:add osd2 d2482f9a-6ef9-4322-9e82-b5d8b8810499 2020-03-11T22:41:24.209 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQAUaWleSU9TDBAAiqHJ7AAmnN3yEQSYhNpAWw== 2020-03-11T22:41:24.209 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQAUaWleSU9TDBAAiqHJ7AAmnN3yEQSYhNpAWw=="}' 2020-03-11T22:41:24.210 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new d2482f9a-6ef9-4322-9e82-b5d8b8810499 -i td/divergent-priors/2/new.json 2020-03-11T22:41:24.676 INFO:tasks.workunit.client.0.smithi059.stdout:2 2020-03-11T22:41:24.686 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/2/new.json 2020-03-11T22:41:24.687 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=cb7f2e42-1934-4145-a3c3-29bdd93f8adb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAUaWleSU9TDBAAiqHJ7AAmnN3yEQSYhNpAWw== --osd-uuid d2482f9a-6ef9-4322-9e82-b5d8b8810499 2020-03-11T22:41:24.717 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:41:24.714+0000 7fed162a3ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:41:24.732 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:41:24.729+0000 7fed162a3ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:41:24.733 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:41:24.731+0000 7fed162a3ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:41:24.734 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:41:24.732+0000 7fed162a3ec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:41:24.734 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:41:24.732+0000 7fed162a3ec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:41:24.734 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:41:24.732+0000 7fed162a3ec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:41:24.798 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:41:24.795+0000 7fed162a3ec0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2020-03-11T22:41:28.140 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/2/keyring 2020-03-11T22:41:28.141 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:41:28.142 INFO:tasks.workunit.client.0.smithi059.stdout:adding osd2 key to auth repository 2020-03-11T22:41:28.142 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-03-11T22:41:28.142 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:41:28.634 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.2 2020-03-11T22:41:28.634 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-03-11T22:41:28.635 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=cb7f2e42-1934-4145-a3c3-29bdd93f8adb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:41:28.635 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:41:28.635 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:41:28.635 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:41:28.662 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:41:28.659+0000 7f26c10f4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:41:28.664 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:41:28.661+0000 7f26c10f4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:41:28.665 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:41:28.662+0000 7f26c10f4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:41:28.998 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-03-11T22:41:28.998 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:41:28.999 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:41:28.999 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:41:28.999 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:41:28.999 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:41:28.999 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:41:28.999 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:41:28.999 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:41:29.000 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:41:29.182 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:41:29.179+0000 7f26c10f4ec0 -1 Falling back to public interface 2020-03-11T22:41:29.396 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:41:29.908 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:41:29.905+0000 7f26c10f4ec0 -1 osd.2 0 log_to_monitors {default=true} 2020-03-11T22:41:30.398 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:41:30.398 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:41:30.398 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:41:30.399 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:41:30.399 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:41:30.399 INFO:tasks.workunit.client.0.smithi059.stdout:1 2020-03-11T22:41:30.779 INFO:tasks.workunit.client.0.smithi059.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/44768,v1:127.0.0.1:6819/44768] [v2:127.0.0.1:6820/44768,v1:127.0.0.1:6821/44768] exists,up d2482f9a-6ef9-4322-9e82-b5d8b8810499 2020-03-11T22:41:30.779 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:41:30.779 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:41:30.780 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:41:30.780 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:243: TEST_divergent_ec: ceph osd set noout 2020-03-11T22:41:31.402 INFO:tasks.workunit.client.0.smithi059.stderr:noout is set 2020-03-11T22:41:31.410 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:244: TEST_divergent_ec: ceph osd set noin 2020-03-11T22:41:32.095 INFO:tasks.workunit.client.0.smithi059.stderr:noin is set 2020-03-11T22:41:32.103 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:245: TEST_divergent_ec: ceph osd set nodown 2020-03-11T22:41:32.913 INFO:tasks.workunit.client.0.smithi059.stderr:nodown is set 2020-03-11T22:41:32.922 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:246: TEST_divergent_ec: create_ec_pool test true k=2 m=1 2020-03-11T22:41:32.922 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: create_ec_pool: local pool_name=test 2020-03-11T22:41:32.922 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: create_ec_pool: shift 2020-03-11T22:41:32.922 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: create_ec_pool: local allow_overwrites=true 2020-03-11T22:41:32.922 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: create_ec_pool: shift 2020-03-11T22:41:32.922 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2020-03-11T22:41:33.470 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: create_ec_pool: create_pool test 1 1 erasure myprofile 2020-03-11T22:41:33.470 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test 1 1 erasure myprofile 2020-03-11T22:41:34.292 INFO:tasks.workunit.client.0.smithi059.stderr:pool 'test' already exists 2020-03-11T22:41:34.299 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:41:35.301 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: create_ec_pool: '[' true = true ']' 2020-03-11T22:41:35.301 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: create_ec_pool: ceph osd pool set test allow_ec_overwrites true 2020-03-11T22:41:36.095 INFO:tasks.workunit.client.0.smithi059.stderr:set pool 1 allow_ec_overwrites to true 2020-03-11T22:41:36.103 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: create_ec_pool: wait_for_clean 2020-03-11T22:41:36.103 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:41:36.104 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:41:36.104 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:41:36.104 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:41:36.104 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:41:36.105 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:41:36.105 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:41:36.105 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:41:36.105 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:41:36.106 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:41:36.200 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:41:36.200 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:41:36.200 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:41:36.201 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:41:36.201 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:41:36.575 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:41:36.575 INFO:tasks.workunit.client.0.smithi059.stderr:1 2020-03-11T22:41:36.576 INFO:tasks.workunit.client.0.smithi059.stderr:2' 2020-03-11T22:41:36.576 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:41:36.576 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:41:36.576 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:41:36.711 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836486 2020-03-11T22:41:36.712 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486' 2020-03-11T22:41:36.712 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:41:36.712 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:41:36.847 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672965 2020-03-11T22:41:36.848 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486 1-42949672965' 2020-03-11T22:41:36.848 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:41:36.848 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:41:36.984 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542148 2020-03-11T22:41:36.984 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486 1-42949672965 2-60129542148' 2020-03-11T22:41:36.984 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:41:36.985 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836486 2020-03-11T22:41:36.985 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:41:36.985 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:41:36.986 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836486 2020-03-11T22:41:36.986 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:41:36.987 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836486 2020-03-11T22:41:36.987 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2020-03-11T22:41:36.987 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:41:36.988 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.0 seq 21474836486 2020-03-11T22:41:37.350 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836486 -lt 21474836486 2020-03-11T22:41:37.350 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:41:37.351 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672965 2020-03-11T22:41:37.351 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:41:37.352 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:41:37.352 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672965 2020-03-11T22:41:37.353 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:41:37.353 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672965 2020-03-11T22:41:37.354 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2020-03-11T22:41:37.354 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:41:37.354 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.1 seq 42949672965 2020-03-11T22:41:37.721 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672965 -lt 42949672965 2020-03-11T22:41:37.722 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:41:37.722 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-60129542148 2020-03-11T22:41:37.722 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:41:37.723 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:41:37.723 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-60129542148 2020-03-11T22:41:37.723 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:41:37.724 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=60129542148 2020-03-11T22:41:37.724 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 60129542148' 2020-03-11T22:41:37.725 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:41:37.725 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.2 seq 60129542148 2020-03-11T22:41:38.088 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 60129542148 -lt 60129542148 2020-03-11T22:41:38.121 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:41:38.121 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:41:38.121 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:41:38.560 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:41:38.560 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:41:38.561 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:41:38.561 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:41:38.562 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:41:38.562 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:41:38.562 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:41:38.562 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:41:38.917 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:41:38.917 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:41:38.917 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:41:38.917 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:41:39.387 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:41:39.388 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:41:39.388 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:41:39.388 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: create_ec_pool: return 0 2020-03-11T22:41:39.388 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:248: TEST_divergent_ec: flush_pg_stats 2020-03-11T22:41:39.388 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:41:39.388 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:41:39.749 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:41:39.749 INFO:tasks.workunit.client.0.smithi059.stderr:1 2020-03-11T22:41:39.749 INFO:tasks.workunit.client.0.smithi059.stderr:2' 2020-03-11T22:41:39.749 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:41:39.750 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:41:39.750 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:41:39.897 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836489 2020-03-11T22:41:39.897 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836489' 2020-03-11T22:41:39.897 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:41:39.898 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:41:40.039 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672968 2020-03-11T22:41:40.039 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836489 1-42949672968' 2020-03-11T22:41:40.040 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:41:40.040 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:41:40.175 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542150 2020-03-11T22:41:40.175 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836489 1-42949672968 2-60129542150' 2020-03-11T22:41:40.175 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:41:40.176 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836489 2020-03-11T22:41:40.176 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:41:40.176 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:41:40.177 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836489 2020-03-11T22:41:40.177 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:41:40.178 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836489 2020-03-11T22:41:40.178 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2020-03-11T22:41:40.178 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:41:40.179 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.0 seq 21474836489 2020-03-11T22:41:40.539 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836487 -lt 21474836489 2020-03-11T22:41:40.539 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:41:41.540 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:41:41.541 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:41:41.943 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836489 -lt 21474836489 2020-03-11T22:41:41.943 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:41:41.944 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672968 2020-03-11T22:41:41.944 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:41:41.944 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:41:41.945 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672968 2020-03-11T22:41:41.945 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:41:41.946 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672968 2020-03-11T22:41:41.946 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2020-03-11T22:41:41.946 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:41:41.946 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.1 seq 42949672968 2020-03-11T22:41:42.312 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672968 -lt 42949672968 2020-03-11T22:41:42.312 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:41:42.313 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-60129542150 2020-03-11T22:41:42.313 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:41:42.313 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:41:42.314 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-60129542150 2020-03-11T22:41:42.314 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:41:42.315 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=60129542150 2020-03-11T22:41:42.315 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 60129542150' 2020-03-11T22:41:42.315 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:41:42.315 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.2 seq 60129542150 2020-03-11T22:41:42.689 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 60129542150 -lt 60129542150 2020-03-11T22:41:42.689 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:249: TEST_divergent_ec: wait_for_clean 2020-03-11T22:41:42.689 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:41:42.689 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:41:42.689 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:41:42.689 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:41:42.690 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:41:42.690 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:41:42.690 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:41:42.691 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:41:42.691 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:41:42.691 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:41:42.786 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:41:42.786 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:41:42.786 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:41:42.787 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:41:42.787 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:41:43.151 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:41:43.151 INFO:tasks.workunit.client.0.smithi059.stderr:1 2020-03-11T22:41:43.151 INFO:tasks.workunit.client.0.smithi059.stderr:2' 2020-03-11T22:41:43.151 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:41:43.152 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:41:43.152 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:41:43.292 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836492 2020-03-11T22:41:43.292 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836492' 2020-03-11T22:41:43.292 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:41:43.292 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:41:43.427 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672970 2020-03-11T22:41:43.428 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836492 1-42949672970' 2020-03-11T22:41:43.428 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:41:43.428 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:41:43.563 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542153 2020-03-11T22:41:43.563 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836492 1-42949672970 2-60129542153' 2020-03-11T22:41:43.563 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:41:43.564 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836492 2020-03-11T22:41:43.564 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:41:43.564 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:41:43.565 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836492 2020-03-11T22:41:43.565 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:41:43.566 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836492 2020-03-11T22:41:43.566 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2020-03-11T22:41:43.566 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:41:43.567 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.0 seq 21474836492 2020-03-11T22:41:43.932 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836489 -lt 21474836492 2020-03-11T22:41:43.932 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:41:44.933 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:41:44.933 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:41:45.311 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836492 -lt 21474836492 2020-03-11T22:41:45.312 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:41:45.312 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672970 2020-03-11T22:41:45.312 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:41:45.313 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:41:45.313 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672970 2020-03-11T22:41:45.314 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:41:45.314 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672970 2020-03-11T22:41:45.315 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2020-03-11T22:41:45.315 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.1 seq 42949672970 2020-03-11T22:41:45.315 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:41:45.685 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672971 -lt 42949672970 2020-03-11T22:41:45.686 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:41:45.686 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-60129542153 2020-03-11T22:41:45.686 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:41:45.687 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:41:45.687 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-60129542153 2020-03-11T22:41:45.688 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:41:45.689 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=60129542153 2020-03-11T22:41:45.689 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 60129542153' 2020-03-11T22:41:45.689 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:41:45.689 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.2 seq 60129542153 2020-03-11T22:41:46.057 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 60129542153 -lt 60129542153 2020-03-11T22:41:46.058 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:41:46.058 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:41:46.058 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:41:46.533 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:41:46.533 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:41:46.534 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:41:46.534 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:41:46.534 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:41:46.534 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:41:46.535 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:41:46.535 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:41:46.882 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:41:46.883 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:41:46.883 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:41:46.883 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:41:47.355 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:41:47.355 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:41:47.356 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:41:47.356 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: ceph pg dump pgs --format=json 2020-03-11T22:41:47.356 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: jq '.pg_stats[0].up_primary' 2020-03-11T22:41:47.699 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2020-03-11T22:41:47.707 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: local divergent=1 2020-03-11T22:41:47.708 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:253: TEST_divergent_ec: echo 'primary and soon to be divergent is 1' 2020-03-11T22:41:47.708 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:254: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:41:47.708 INFO:tasks.workunit.client.0.smithi059.stdout:primary and soon to be divergent is 1 2020-03-11T22:41:48.049 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:41:48.049 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:41:35.011862+0000 0'0 26:11 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:33.943641+0000 0'0 2020-03-11T22:41:33.943641+0000 0 2020-03-11T22:41:48.049 INFO:tasks.workunit.client.0.smithi059.stdout: 2020-03-11T22:41:48.049 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:41:48.050 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2020-03-11T22:41:48.059 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:255: TEST_divergent_ec: local non_divergent= 2020-03-11T22:41:48.060 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2020-03-11T22:41:48.060 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 0 = 1 ']' 2020-03-11T22:41:48.060 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:261: TEST_divergent_ec: non_divergent=' 0' 2020-03-11T22:41:48.060 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2020-03-11T22:41:48.060 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 1 = 1 ']' 2020-03-11T22:41:48.060 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:259: TEST_divergent_ec: continue 2020-03-11T22:41:48.061 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2020-03-11T22:41:48.061 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 2 = 1 ']' 2020-03-11T22:41:48.062 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:261: TEST_divergent_ec: non_divergent=' 0 2' 2020-03-11T22:41:48.062 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:264: TEST_divergent_ec: echo 'writing initial objects' 2020-03-11T22:41:48.062 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: seq 1 100 2020-03-11T22:41:48.062 INFO:tasks.workunit.client.0.smithi059.stdout:writing initial objects 2020-03-11T22:41:48.063 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:48.063 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_1 /etc/fstab 2020-03-11T22:41:48.116 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:48.116 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_2 /etc/fstab 2020-03-11T22:41:48.156 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:48.156 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_3 /etc/fstab 2020-03-11T22:41:48.199 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:48.199 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_4 /etc/fstab 2020-03-11T22:41:48.240 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:48.240 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_5 /etc/fstab 2020-03-11T22:41:48.365 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:48.366 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_6 /etc/fstab 2020-03-11T22:41:48.406 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:48.407 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_7 /etc/fstab 2020-03-11T22:41:48.449 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:48.449 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_8 /etc/fstab 2020-03-11T22:41:48.490 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:48.490 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_9 /etc/fstab 2020-03-11T22:41:48.578 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:48.578 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_10 /etc/fstab 2020-03-11T22:41:48.678 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:48.678 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_11 /etc/fstab 2020-03-11T22:41:48.724 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:48.724 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_12 /etc/fstab 2020-03-11T22:41:48.765 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:48.765 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_13 /etc/fstab 2020-03-11T22:41:48.807 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:48.807 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_14 /etc/fstab 2020-03-11T22:41:48.849 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:48.849 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_15 /etc/fstab 2020-03-11T22:41:48.945 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:48.945 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_16 /etc/fstab 2020-03-11T22:41:48.991 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:48.991 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_17 /etc/fstab 2020-03-11T22:41:49.032 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:49.032 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_18 /etc/fstab 2020-03-11T22:41:49.120 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:49.121 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_19 /etc/fstab 2020-03-11T22:41:49.279 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:49.279 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_20 /etc/fstab 2020-03-11T22:41:49.371 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:49.372 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_21 /etc/fstab 2020-03-11T22:41:49.416 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:49.416 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_22 /etc/fstab 2020-03-11T22:41:49.500 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:49.500 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_23 /etc/fstab 2020-03-11T22:41:49.541 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:49.541 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_24 /etc/fstab 2020-03-11T22:41:49.582 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:49.582 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_25 /etc/fstab 2020-03-11T22:41:49.657 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:49.658 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_26 /etc/fstab 2020-03-11T22:41:49.699 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:49.699 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_27 /etc/fstab 2020-03-11T22:41:49.765 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:49.765 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_28 /etc/fstab 2020-03-11T22:41:49.808 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:49.808 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_29 /etc/fstab 2020-03-11T22:41:49.889 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:49.889 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_30 /etc/fstab 2020-03-11T22:41:49.972 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:49.973 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_31 /etc/fstab 2020-03-11T22:41:50.058 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:50.058 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_32 /etc/fstab 2020-03-11T22:41:50.099 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:50.099 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_33 /etc/fstab 2020-03-11T22:41:50.141 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:50.141 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_34 /etc/fstab 2020-03-11T22:41:50.223 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:50.223 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_35 /etc/fstab 2020-03-11T22:41:50.348 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:50.348 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_36 /etc/fstab 2020-03-11T22:41:50.391 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:50.391 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_37 /etc/fstab 2020-03-11T22:41:50.433 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:50.433 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_38 /etc/fstab 2020-03-11T22:41:50.478 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:50.478 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_39 /etc/fstab 2020-03-11T22:41:50.635 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:50.636 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_40 /etc/fstab 2020-03-11T22:41:50.683 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:50.683 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_41 /etc/fstab 2020-03-11T22:41:50.726 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:50.726 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_42 /etc/fstab 2020-03-11T22:41:50.765 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:50.766 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_43 /etc/fstab 2020-03-11T22:41:50.808 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:50.808 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_44 /etc/fstab 2020-03-11T22:41:50.949 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:50.949 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_45 /etc/fstab 2020-03-11T22:41:51.000 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:51.000 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_46 /etc/fstab 2020-03-11T22:41:51.041 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:51.041 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_47 /etc/fstab 2020-03-11T22:41:51.082 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:51.082 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_48 /etc/fstab 2020-03-11T22:41:51.166 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:51.166 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_49 /etc/fstab 2020-03-11T22:41:51.350 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:51.351 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_50 /etc/fstab 2020-03-11T22:41:51.400 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:51.400 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_51 /etc/fstab 2020-03-11T22:41:51.441 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:51.441 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_52 /etc/fstab 2020-03-11T22:41:51.576 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:51.576 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_53 /etc/fstab 2020-03-11T22:41:51.676 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:51.676 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_54 /etc/fstab 2020-03-11T22:41:51.724 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:51.724 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_55 /etc/fstab 2020-03-11T22:41:51.843 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:51.843 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_56 /etc/fstab 2020-03-11T22:41:51.900 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:51.900 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_57 /etc/fstab 2020-03-11T22:41:51.941 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:51.942 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_58 /etc/fstab 2020-03-11T22:41:52.085 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:52.085 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_59 /etc/fstab 2020-03-11T22:41:52.166 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:52.166 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_60 /etc/fstab 2020-03-11T22:41:52.209 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:52.209 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_61 /etc/fstab 2020-03-11T22:41:52.259 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:52.259 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_62 /etc/fstab 2020-03-11T22:41:52.299 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:52.300 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_63 /etc/fstab 2020-03-11T22:41:52.403 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:52.403 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_64 /etc/fstab 2020-03-11T22:41:52.450 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:52.451 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_65 /etc/fstab 2020-03-11T22:41:52.491 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:52.491 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_66 /etc/fstab 2020-03-11T22:41:52.534 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:52.534 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_67 /etc/fstab 2020-03-11T22:41:52.584 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:52.584 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_68 /etc/fstab 2020-03-11T22:41:52.747 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:52.747 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_69 /etc/fstab 2020-03-11T22:41:52.798 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:52.798 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_70 /etc/fstab 2020-03-11T22:41:52.842 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:52.842 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_71 /etc/fstab 2020-03-11T22:41:52.911 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:52.911 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_72 /etc/fstab 2020-03-11T22:41:52.959 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:52.959 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_73 /etc/fstab 2020-03-11T22:41:53.037 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:53.037 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_74 /etc/fstab 2020-03-11T22:41:53.084 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:53.084 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_75 /etc/fstab 2020-03-11T22:41:53.125 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:53.125 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_76 /etc/fstab 2020-03-11T22:41:53.209 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:53.209 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_77 /etc/fstab 2020-03-11T22:41:53.313 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:53.313 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_78 /etc/fstab 2020-03-11T22:41:53.405 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:53.405 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_79 /etc/fstab 2020-03-11T22:41:53.484 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:53.484 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_80 /etc/fstab 2020-03-11T22:41:53.534 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:53.534 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_81 /etc/fstab 2020-03-11T22:41:53.575 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:53.575 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_82 /etc/fstab 2020-03-11T22:41:53.655 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:53.655 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_83 /etc/fstab 2020-03-11T22:41:53.701 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:53.701 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_84 /etc/fstab 2020-03-11T22:41:53.777 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:53.777 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_85 /etc/fstab 2020-03-11T22:41:53.872 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:53.873 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_86 /etc/fstab 2020-03-11T22:41:53.918 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:53.918 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_87 /etc/fstab 2020-03-11T22:41:53.997 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:53.998 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_88 /etc/fstab 2020-03-11T22:41:54.108 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:54.108 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_89 /etc/fstab 2020-03-11T22:41:54.151 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:54.151 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_90 /etc/fstab 2020-03-11T22:41:54.194 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:54.194 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_91 /etc/fstab 2020-03-11T22:41:54.234 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:54.235 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_92 /etc/fstab 2020-03-11T22:41:54.357 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:54.357 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_93 /etc/fstab 2020-03-11T22:41:54.401 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:54.401 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_94 /etc/fstab 2020-03-11T22:41:54.442 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:54.442 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_95 /etc/fstab 2020-03-11T22:41:54.484 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:54.485 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_96 /etc/fstab 2020-03-11T22:41:54.525 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:54.526 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_97 /etc/fstab 2020-03-11T22:41:54.674 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:54.674 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_98 /etc/fstab 2020-03-11T22:41:54.718 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:54.718 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_99 /etc/fstab 2020-03-11T22:41:54.759 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:41:54.759 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_100 /etc/fstab 2020-03-11T22:41:54.803 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:271: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:41:54.803 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:271: TEST_divergent_ec: wait_for_clean 2020-03-11T22:41:54.803 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:41:54.804 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:41:54.804 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:41:54.804 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:41:54.804 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:41:54.805 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:41:54.805 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:41:54.805 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:41:54.805 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:41:54.805 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:41:54.862 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:41:54.862 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:41:54.862 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:41:54.862 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:41:54.862 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:41:55.249 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:41:55.249 INFO:tasks.workunit.client.0.smithi059.stderr:1 2020-03-11T22:41:55.250 INFO:tasks.workunit.client.0.smithi059.stderr:2' 2020-03-11T22:41:55.250 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:41:55.250 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:41:55.250 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:41:55.391 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836496 2020-03-11T22:41:55.392 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836496' 2020-03-11T22:41:55.392 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:41:55.392 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:41:55.523 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672975 2020-03-11T22:41:55.523 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836496 1-42949672975' 2020-03-11T22:41:55.523 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:41:55.523 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:41:55.659 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542157 2020-03-11T22:41:55.659 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836496 1-42949672975 2-60129542157' 2020-03-11T22:41:55.659 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:41:55.660 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836496 2020-03-11T22:41:55.660 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:41:55.660 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:41:55.661 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836496 2020-03-11T22:41:55.661 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:41:55.662 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836496 2020-03-11T22:41:55.662 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836496' 2020-03-11T22:41:55.662 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:41:55.662 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.0 seq 21474836496 2020-03-11T22:41:56.032 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836494 -lt 21474836496 2020-03-11T22:41:56.032 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:41:57.033 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:41:57.033 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:41:57.410 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836496 -lt 21474836496 2020-03-11T22:41:57.410 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:41:57.410 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672975 2020-03-11T22:41:57.411 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:41:57.411 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:41:57.412 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672975 2020-03-11T22:41:57.412 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:41:57.413 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672975 2020-03-11T22:41:57.413 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2020-03-11T22:41:57.413 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:41:57.413 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.1 seq 42949672975 2020-03-11T22:41:57.782 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672975 -lt 42949672975 2020-03-11T22:41:57.782 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:41:57.783 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-60129542157 2020-03-11T22:41:57.783 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:41:57.783 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:41:57.784 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-60129542157 2020-03-11T22:41:57.784 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:41:57.785 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=60129542157 2020-03-11T22:41:57.785 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 60129542157' 2020-03-11T22:41:57.785 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:41:57.786 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.2 seq 60129542157 2020-03-11T22:41:58.166 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 60129542158 -lt 60129542157 2020-03-11T22:41:58.166 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:41:58.166 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:41:58.167 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:41:58.643 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:41:58.643 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:41:58.643 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:41:58.643 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:41:58.644 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:41:58.644 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:41:58.644 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:41:58.644 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:41:58.993 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:41:58.993 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:41:58.993 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:41:58.994 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:41:59.478 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:41:59.478 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:41:59.478 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:41:59.479 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:273: TEST_divergent_ec: get_pg test existing_1 2020-03-11T22:41:59.479 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1030: get_pg: local poolname=test 2020-03-11T22:41:59.480 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local objectname=existing_1 2020-03-11T22:41:59.480 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: ceph --format json osd map test existing_1 2020-03-11T22:41:59.480 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: jq -r .pgid 2020-03-11T22:41:59.845 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:273: TEST_divergent_ec: local pgid=1.0 2020-03-11T22:41:59.846 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:276: TEST_divergent_ec: echo 'blackholing osds 0 2' 2020-03-11T22:41:59.846 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:277: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:41:59.847 INFO:tasks.workunit.client.0.smithi059.stdout:blackholing osds 0 2 2020-03-11T22:42:00.231 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:42:00.231 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 100 0 0 0 0 48300 0 0 16 16 active+clean 2020-03-11T22:41:35.011862+0000 28'100 28:111 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:33.943641+0000 0'0 2020-03-11T22:41:33.943641+0000 0 2020-03-11T22:42:00.231 INFO:tasks.workunit.client.0.smithi059.stdout: 2020-03-11T22:42:00.231 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:42:00.232 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2020-03-11T22:42:00.241 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:278: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:42:00.242 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: get_asok_path osd.0 2020-03-11T22:42:00.242 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-11T22:42:00.242 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-11T22:42:00.242 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:42:00.242 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:42:00.243 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:42:00.243 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17931/ceph-osd.0.asok 2020-03-11T22:42:00.243 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: CEPH_ARGS= 2020-03-11T22:42:00.243 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.17931/ceph-osd.0.asok config set objectstore_blackhole 1 2020-03-11T22:42:00.333 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2020-03-11T22:42:00.333 INFO:tasks.workunit.client.0.smithi059.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-03-11T22:42:00.333 INFO:tasks.workunit.client.0.smithi059.stdout:} 2020-03-11T22:42:00.340 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:278: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:42:00.341 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: get_asok_path osd.2 2020-03-11T22:42:00.341 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-03-11T22:42:00.341 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-03-11T22:42:00.342 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:42:00.342 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:42:00.342 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:42:00.342 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17931/ceph-osd.2.asok 2020-03-11T22:42:00.342 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: CEPH_ARGS= 2020-03-11T22:42:00.342 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.17931/ceph-osd.2.asok config set objectstore_blackhole 1 2020-03-11T22:42:00.432 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2020-03-11T22:42:00.432 INFO:tasks.workunit.client.0.smithi059.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-03-11T22:42:00.433 INFO:tasks.workunit.client.0.smithi059.stdout:} 2020-03-11T22:42:00.440 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:284: TEST_divergent_ec: echo 'writing divergent object' 2020-03-11T22:42:00.440 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:286: TEST_divergent_ec: sleep 1 2020-03-11T22:42:00.440 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:285: TEST_divergent_ec: rados -p test put existing_100 /etc/resolv.conf 2020-03-11T22:42:00.440 INFO:tasks.workunit.client.0.smithi059.stdout:writing divergent object 2020-03-11T22:42:01.441 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:288: TEST_divergent_ec: rados -p test mksnap snap1 2020-03-11T22:42:01.442 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:287: TEST_divergent_ec: rados -p test put existing_100 /etc/fstab 2020-03-11T22:42:01.603 INFO:tasks.workunit.client.0.smithi059.stdout:created pool test snap snap1 2020-03-11T22:42:01.605 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:290: TEST_divergent_ec: sleep 10 2020-03-11T22:42:01.606 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:289: TEST_divergent_ec: expr 100 - 1 2020-03-11T22:42:01.607 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:289: TEST_divergent_ec: rados -p test put existing_99 /etc/fstab 2020-03-11T22:42:11.607 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:291: TEST_divergent_ec: killall -9 rados 2020-03-11T22:42:11.612 INFO:tasks.workunit.client.0.smithi059.stderr:rados: no process found 2020-03-11T22:42:11.612 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:294: TEST_divergent_ec: echo 'killing all the osds' 2020-03-11T22:42:11.612 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:295: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:42:11.612 INFO:tasks.workunit.client.0.smithi059.stdout:killing all the osds 2020-03-11T22:42:11.967 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:42:11.968 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 101 0 0 0 0 48783 0 0 14 14 active+clean 2020-03-11T22:41:35.011862+0000 29'104 29:115 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:33.943641+0000 0'0 2020-03-11T22:41:33.943641+0000 0 2020-03-11T22:42:11.968 INFO:tasks.workunit.client.0.smithi059.stdout: 2020-03-11T22:42:11.968 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:42:11.968 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2020-03-11T22:42:11.974 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:296: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd 2020-03-11T22:42:11.974 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:42:11.974 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:42:11.975 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:42:11.975 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:42:11.975 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:42:12.081 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:42:12.081 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2020-03-11T22:42:12.081 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.0 2020-03-11T22:42:12.789 INFO:tasks.workunit.client.0.smithi059.stderr:osd.0 is already down. 2020-03-11T22:42:12.797 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2020-03-11T22:42:12.797 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.1 2020-03-11T22:42:13.355 INFO:tasks.workunit.client.0.smithi059.stderr:osd.1 is already down. 2020-03-11T22:42:13.363 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2020-03-11T22:42:13.363 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.2 2020-03-11T22:42:13.916 INFO:tasks.workunit.client.0.smithi059.stderr:osd.2 is already down. 2020-03-11T22:42:13.924 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:301: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:42:13.924 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:303: TEST_divergent_ec: ceph osd out osd.0 2020-03-11T22:42:14.482 INFO:tasks.workunit.client.0.smithi059.stderr:osd.0 is already out. 2020-03-11T22:42:14.490 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:301: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:42:14.491 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:303: TEST_divergent_ec: ceph osd out osd.2 2020-03-11T22:42:15.051 INFO:tasks.workunit.client.0.smithi059.stderr:osd.2 is already out. 2020-03-11T22:42:15.059 INFO:tasks.workunit.client.0.smithi059.stdout:bringing up non_divergent 0 2 2020-03-11T22:42:15.059 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:307: TEST_divergent_ec: echo 'bringing up non_divergent 0 2' 2020-03-11T22:42:15.059 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:308: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:42:15.502 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:42:15.502 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 101 0 0 0 0 48783 0 0 14 14 stale+active+clean 2020-03-11T22:41:35.011862+0000 29'104 29:115 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:33.943641+0000 0'0 2020-03-11T22:41:33.943641+0000 0 2020-03-11T22:42:15.502 INFO:tasks.workunit.client.0.smithi059.stdout: 2020-03-11T22:42:15.502 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:42:15.504 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2020-03-11T22:42:15.515 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:309: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:42:15.515 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:311: TEST_divergent_ec: activate_osd td/divergent-priors 0 2020-03-11T22:42:15.515 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:42:15.515 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:42:15.516 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=0 2020-03-11T22:42:15.516 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:42:15.516 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/0 2020-03-11T22:42:15.516 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=cb7f2e42-1934-4145-a3c3-29bdd93f8adb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:42:15.516 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:42:15.516 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:42:15.516 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:42:15.517 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-11T22:42:15.517 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-11T22:42:15.517 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:42:15.517 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:42:15.518 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:42:15.518 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:42:15.518 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:42:15.518 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:42:15.519 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:42:15.519 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:42:15.519 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:42:15.519 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17931/$cluster-$name.asok' 2020-03-11T22:42:15.519 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' 2020-03-11T22:42:15.519 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:42:15.520 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:42:15.520 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:42:15.520 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:42:15.520 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:42:15.520 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:42:15.520 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:42:15.521 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:42:15.521 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/0 2020-03-11T22:42:15.521 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.0 2020-03-11T22:42:15.521 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 0 --fsid=cb7f2e42-1934-4145-a3c3-29bdd93f8adb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:42:15.521 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/0/whoami 2020-03-11T22:42:15.521 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.0 2020-03-11T22:42:15.522 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 0 = 0 ']' 2020-03-11T22:42:15.522 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:42:15.522 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:42:15.523 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:42:15.543 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:42:15.539+0000 7f061739cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:15.546 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:42:15.541+0000 7f061739cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:15.546 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:42:15.542+0000 7f061739cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:15.950 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 0 2020-03-11T22:42:15.951 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:42:15.951 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:42:15.951 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:42:15.951 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:42:15.951 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:42:15.951 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:42:15.951 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:42:15.952 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:42:15.952 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:42:16.068 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:42:16.064+0000 7f061739cec0 -1 Falling back to public interface 2020-03-11T22:42:16.467 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:42:16.955 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:42:16.952+0000 7f061739cec0 -1 osd.0 28 log_to_monitors {default=true} 2020-03-11T22:42:17.468 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:42:17.469 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:42:17.469 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:42:17.469 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:42:17.469 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:42:17.469 INFO:tasks.workunit.client.0.smithi059.stdout:1 2020-03-11T22:42:17.854 INFO:tasks.workunit.client.0.smithi059.stdout:osd.0 up out weight 0 up_from 35 up_thru 0 down_at 30 last_clean_interval [5,28) [v2:127.0.0.1:6802/49144,v1:127.0.0.1:6803/49144] [v2:127.0.0.1:6804/49144,v1:127.0.0.1:6805/49144] exists,up af7f49a6-1c6c-446f-802f-d448ee247966 2020-03-11T22:42:17.855 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.2 2020-03-11T22:42:17.855 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:42:17.855 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:42:17.855 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:42:17.855 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:309: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:42:17.856 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:311: TEST_divergent_ec: activate_osd td/divergent-priors 2 2020-03-11T22:42:17.856 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:42:17.856 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:42:17.856 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=2 2020-03-11T22:42:17.856 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:42:17.856 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/2 2020-03-11T22:42:17.856 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=cb7f2e42-1934-4145-a3c3-29bdd93f8adb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:42:17.857 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:42:17.857 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:42:17.857 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:42:17.857 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-11T22:42:17.857 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-11T22:42:17.857 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:42:17.858 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:42:17.858 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:42:17.858 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:42:17.858 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:42:17.858 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:42:17.859 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:42:17.859 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:42:17.859 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:42:17.859 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17931/$cluster-$name.asok' 2020-03-11T22:42:17.859 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' 2020-03-11T22:42:17.859 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:42:17.859 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:42:17.860 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:42:17.860 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:42:17.860 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:42:17.860 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:42:17.860 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:42:17.860 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:42:17.860 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/2 2020-03-11T22:42:17.861 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.2 2020-03-11T22:42:17.861 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 2 --fsid=cb7f2e42-1934-4145-a3c3-29bdd93f8adb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:42:17.861 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/2/whoami 2020-03-11T22:42:17.861 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 2 = 2 ']' 2020-03-11T22:42:17.862 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:42:17.862 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:42:17.863 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:42:17.881 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:42:17.878+0000 7f56c15c9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:17.883 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:42:17.880+0000 7f56c15c9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:17.884 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:42:17.881+0000 7f56c15c9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:18.226 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 2 2020-03-11T22:42:18.227 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:42:18.227 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:42:18.227 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:42:18.227 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:42:18.227 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:42:18.227 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:42:18.227 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:42:18.228 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:42:18.228 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:42:18.402 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:42:18.399+0000 7f56c15c9ec0 -1 Falling back to public interface 2020-03-11T22:42:18.589 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:42:19.311 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:42:19.307+0000 7f56c15c9ec0 -1 osd.2 28 log_to_monitors {default=true} 2020-03-11T22:42:19.590 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:42:19.590 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:42:19.591 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:42:19.591 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:42:19.591 INFO:tasks.workunit.client.0.smithi059.stdout:1 2020-03-11T22:42:19.591 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:42:19.969 INFO:tasks.workunit.client.0.smithi059.stdout:osd.2 up out weight 0 up_from 36 up_thru 0 down_at 32 last_clean_interval [14,28) [v2:127.0.0.1:6810/49338,v1:127.0.0.1:6811/49338] [v2:127.0.0.1:6812/49338,v1:127.0.0.1:6813/49338] exists,up d2482f9a-6ef9-4322-9e82-b5d8b8810499 2020-03-11T22:42:19.969 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:42:19.969 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:42:19.969 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:42:19.969 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:313: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:42:19.969 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:315: TEST_divergent_ec: ceph osd in osd.0 2020-03-11T22:42:20.461 INFO:tasks.workunit.client.0.smithi059.stderr:osd.0 is already in. 2020-03-11T22:42:20.468 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:313: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:42:20.469 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:315: TEST_divergent_ec: ceph osd in osd.2 2020-03-11T22:42:20.970 INFO:tasks.workunit.client.0.smithi059.stderr:osd.2 is already in. 2020-03-11T22:42:20.978 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:318: TEST_divergent_ec: sleep 5 2020-03-11T22:42:25.980 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:322: TEST_divergent_ec: expr 2 + 2 2020-03-11T22:42:25.981 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:322: TEST_divergent_ec: objname=existing_4 2020-03-11T22:42:25.981 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:323: TEST_divergent_ec: echo 'writing non-divergent object existing_4' 2020-03-11T22:42:25.981 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:324: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:42:25.981 INFO:tasks.workunit.client.0.smithi059.stdout:writing non-divergent object existing_4 2020-03-11T22:42:26.337 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:42:26.337 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 100 0 100 0 0 48300 0 0 16 16 active+undersized+degraded 2020-03-11T22:42:21.148558+0000 28'100 39:120 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2020-03-11T22:41:33.943641+0000 0'0 2020-03-11T22:41:33.943641+0000 0 2020-03-11T22:42:26.338 INFO:tasks.workunit.client.0.smithi059.stdout: 2020-03-11T22:42:26.338 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:42:26.338 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2020-03-11T22:42:26.346 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:325: TEST_divergent_ec: rados -p test put existing_4 /etc/resolv.conf 2020-03-11T22:42:26.428 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:327: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:42:26.428 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:327: TEST_divergent_ec: wait_for_clean 2020-03-11T22:42:26.428 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:42:26.428 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:42:26.428 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:42:26.428 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:42:26.429 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:42:26.429 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:42:26.430 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:42:26.430 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:42:26.430 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:42:26.430 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:42:26.488 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:42:26.488 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:42:26.488 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:42:26.488 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:42:26.489 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:42:26.854 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:42:26.854 INFO:tasks.workunit.client.0.smithi059.stderr:1 2020-03-11T22:42:26.854 INFO:tasks.workunit.client.0.smithi059.stderr:2' 2020-03-11T22:42:26.855 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:42:26.855 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:42:26.855 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:42:27.012 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=150323855365 2020-03-11T22:42:27.012 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-150323855365' 2020-03-11T22:42:27.013 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:42:27.013 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:42:27.108 INFO:tasks.workunit.client.0.smithi059.stderr:Error ENXIO: problem getting command descriptions from osd.1 2020-03-11T22:42:27.116 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq= 2020-03-11T22:42:27.116 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-150323855365 1-' 2020-03-11T22:42:27.116 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:42:27.117 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:42:27.255 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=154618822660 2020-03-11T22:42:27.255 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-150323855365 1- 2-154618822660' 2020-03-11T22:42:27.255 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:42:27.255 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-150323855365 2020-03-11T22:42:27.255 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:42:27.256 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:42:27.256 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-150323855365 2020-03-11T22:42:27.256 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:42:27.257 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=150323855365 2020-03-11T22:42:27.257 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 150323855365' 2020-03-11T22:42:27.258 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.0 seq 150323855365 2020-03-11T22:42:27.258 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:42:27.621 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 150323855365 -lt 150323855365 2020-03-11T22:42:27.621 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:42:27.622 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1- 2020-03-11T22:42:27.622 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:42:27.622 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:42:27.623 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1- 2020-03-11T22:42:27.623 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:42:27.624 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq= 2020-03-11T22:42:27.624 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq ' 2020-03-11T22:42:27.624 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.1 seq 2020-03-11T22:42:27.624 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:42:28.028 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672978 -lt 2020-03-11T22:42:28.028 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 2079: test: 42949672978: unary operator expected 2020-03-11T22:42:28.028 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:42:28.029 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-154618822660 2020-03-11T22:42:28.029 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:42:28.030 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:42:28.030 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-154618822660 2020-03-11T22:42:28.030 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:42:28.031 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=154618822660 2020-03-11T22:42:28.031 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 154618822660' 2020-03-11T22:42:28.032 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:42:28.032 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.2 seq 154618822660 2020-03-11T22:42:28.404 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 154618822658 -lt 154618822660 2020-03-11T22:42:28.405 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:42:29.405 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:42:29.406 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:42:29.783 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 154618822660 -lt 154618822660 2020-03-11T22:42:29.798 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:42:29.798 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:42:29.798 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:42:30.253 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:42:30.253 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:42:30.254 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:42:30.254 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:42:30.254 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:42:30.254 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:42:30.254 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:42:30.254 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:42:30.605 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:42:30.606 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:42:30.606 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:42:30.606 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:42:31.081 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:42:31.082 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:42:31.082 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:42:31.082 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:42:31.082 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:42:31.082 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:42:31.183 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:42:31.183 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:42:31.183 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:42:31.183 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:42:31.183 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:42:31.184 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:42:31.184 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:42:31.184 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:42:31.549 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:42:31.549 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:42:31.549 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:42:31.550 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:42:32.023 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:42:32.023 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:42:32.024 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:42:32.024 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:42:32.024 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:42:32.024 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:42:32.024 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:42:32.024 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:42:32.024 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:42:32.496 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:42:32.496 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:42:32.496 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 1 >= 8 )) 2020-03-11T22:42:32.496 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:42:32.496 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.2 2020-03-11T22:42:32.697 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:42:32.697 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:42:32.697 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:42:32.698 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:42:32.698 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:42:32.698 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:42:32.698 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:42:32.698 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:42:33.050 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:42:33.051 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:42:33.051 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:42:33.051 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:42:33.526 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:42:33.526 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:42:33.527 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:42:33.527 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:42:33.527 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:42:33.527 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:42:33.527 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:42:33.528 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:42:33.528 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:42:34.004 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:42:34.005 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:42:34.005 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 2 >= 8 )) 2020-03-11T22:42:34.005 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:42:34.005 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.4 2020-03-11T22:42:34.401 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:42:34.401 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:42:34.402 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:42:34.402 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:42:34.402 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:42:34.402 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:42:34.403 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:42:34.403 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:42:34.758 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:42:34.759 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:42:34.759 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:42:34.759 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:42:35.228 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:42:35.229 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:42:35.229 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:42:35.229 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:42:35.229 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:42:35.229 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:42:35.229 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:42:35.229 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:42:35.230 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:42:35.700 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:42:35.700 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:42:35.700 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 3 >= 8 )) 2020-03-11T22:42:35.701 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:42:35.701 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.8 2020-03-11T22:42:36.501 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:42:36.501 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:42:36.502 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:42:36.503 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:42:36.503 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:42:36.503 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:42:36.504 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:42:36.504 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:42:36.865 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:42:36.866 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:42:36.866 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:42:36.866 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:42:37.335 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:42:37.335 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:42:37.336 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:42:37.336 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:42:37.336 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:42:37.336 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:42:37.336 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:42:37.336 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:42:37.336 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:42:37.828 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:42:37.828 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:42:37.828 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 4 >= 8 )) 2020-03-11T22:42:37.828 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:42:37.829 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 1.6 2020-03-11T22:42:39.429 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:42:39.429 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:42:39.430 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:42:39.430 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:42:39.430 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:42:39.430 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:42:39.430 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:42:39.430 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:42:39.794 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:42:39.794 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:42:39.794 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:42:39.795 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:42:40.265 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:42:40.266 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:42:40.266 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:42:40.266 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:42:40.266 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:42:40.267 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:42:40.267 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:42:40.267 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:42:40.267 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:42:40.737 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:42:40.737 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:42:40.737 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 5 >= 8 )) 2020-03-11T22:42:40.737 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:42:40.737 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 3.2 2020-03-11T22:42:43.938 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:42:43.938 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:42:43.939 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:42:43.939 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:42:43.939 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:42:43.940 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:42:43.940 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:42:43.940 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:42:44.305 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:42:44.306 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:42:44.306 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:42:44.306 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:42:44.783 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:42:44.784 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:42:44.784 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:42:44.784 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:42:44.784 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:42:44.784 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:42:44.784 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:42:44.784 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:42:44.785 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:42:45.278 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:42:45.278 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:42:45.279 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 6 >= 8 )) 2020-03-11T22:42:45.279 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:42:45.279 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 6.4 2020-03-11T22:42:51.679 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:42:51.679 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:42:51.680 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:42:51.680 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:42:51.680 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:42:51.681 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:42:51.681 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:42:51.681 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:42:52.041 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:42:52.042 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:42:52.042 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:42:52.042 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:42:52.513 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:42:52.513 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:42:52.514 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:42:52.514 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:42:52.514 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:42:52.514 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:42:52.514 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:42:52.514 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:42:52.515 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:42:52.993 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:42:52.993 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:42:52.993 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 7 >= 8 )) 2020-03-11T22:42:52.994 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:42:52.994 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 7.3 2020-03-11T22:43:00.293 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:43:00.293 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:43:00.294 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:43:00.294 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:43:00.294 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:43:00.295 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:43:00.295 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:43:00.295 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:43:00.654 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:43:00.654 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:43:00.655 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:43:00.655 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:43:01.126 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:43:01.126 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:43:01.127 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:43:01.127 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:43:01.127 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:43:01.127 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:43:01.127 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:43:01.127 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:43:01.128 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:43:01.597 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:43:01.597 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:43:01.597 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 8 >= 8 )) 2020-03-11T22:43:01.598 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: ceph report 2020-03-11T22:43:02.062 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2020-03-11T22:43:02.062 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_fingerprint": "c0ec5b4a-1e22-41f2-aed6-87f25ac30cdf", 2020-03-11T22:43:02.062 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "15.1.0-1941-g4acc112", 2020-03-11T22:43:02.062 INFO:tasks.workunit.client.0.smithi059.stdout: "commit": "4acc1121e47039a3cdf2978ae458c10d153aa141", 2020-03-11T22:43:02.062 INFO:tasks.workunit.client.0.smithi059.stdout: "timestamp": "2020-03-11T22:43:02.058669+0000", 2020-03-11T22:43:02.062 INFO:tasks.workunit.client.0.smithi059.stdout: "tag": "", 2020-03-11T22:43:02.063 INFO:tasks.workunit.client.0.smithi059.stdout: "health": { 2020-03-11T22:43:02.063 INFO:tasks.workunit.client.0.smithi059.stdout: "status": "HEALTH_WARN", 2020-03-11T22:43:02.063 INFO:tasks.workunit.client.0.smithi059.stdout: "checks": { 2020-03-11T22:43:02.063 INFO:tasks.workunit.client.0.smithi059.stdout: "OSDMAP_FLAGS": { 2020-03-11T22:43:02.063 INFO:tasks.workunit.client.0.smithi059.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:43:02.063 INFO:tasks.workunit.client.0.smithi059.stdout: "summary": { 2020-03-11T22:43:02.063 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "nodown,noout,noin flag(s) set", 2020-03-11T22:43:02.064 INFO:tasks.workunit.client.0.smithi059.stdout: "count": 17 2020-03-11T22:43:02.064 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.064 INFO:tasks.workunit.client.0.smithi059.stdout: "detail": [], 2020-03-11T22:43:02.064 INFO:tasks.workunit.client.0.smithi059.stdout: "muted": false 2020-03-11T22:43:02.064 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.064 INFO:tasks.workunit.client.0.smithi059.stdout: "OSD_DOWN": { 2020-03-11T22:43:02.064 INFO:tasks.workunit.client.0.smithi059.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:43:02.064 INFO:tasks.workunit.client.0.smithi059.stdout: "summary": { 2020-03-11T22:43:02.065 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "1 osds down", 2020-03-11T22:43:02.065 INFO:tasks.workunit.client.0.smithi059.stdout: "count": 1 2020-03-11T22:43:02.065 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.065 INFO:tasks.workunit.client.0.smithi059.stdout: "detail": [ 2020-03-11T22:43:02.065 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.065 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "osd.1 (root=default,host=smithi059) is down" 2020-03-11T22:43:02.065 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:02.066 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:43:02.066 INFO:tasks.workunit.client.0.smithi059.stdout: "muted": false 2020-03-11T22:43:02.066 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.066 INFO:tasks.workunit.client.0.smithi059.stdout: "PG_DEGRADED": { 2020-03-11T22:43:02.066 INFO:tasks.workunit.client.0.smithi059.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:43:02.066 INFO:tasks.workunit.client.0.smithi059.stdout: "summary": { 2020-03-11T22:43:02.066 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "Degraded data redundancy: 101/303 objects degraded (33.333%), 1 pg degraded", 2020-03-11T22:43:02.067 INFO:tasks.workunit.client.0.smithi059.stdout: "count": 1 2020-03-11T22:43:02.067 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.067 INFO:tasks.workunit.client.0.smithi059.stdout: "detail": [ 2020-03-11T22:43:02.067 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.067 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [2147483647,0,2]" 2020-03-11T22:43:02.067 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:02.067 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:43:02.067 INFO:tasks.workunit.client.0.smithi059.stdout: "muted": false 2020-03-11T22:43:02.068 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.068 INFO:tasks.workunit.client.0.smithi059.stdout: "POOL_APP_NOT_ENABLED": { 2020-03-11T22:43:02.068 INFO:tasks.workunit.client.0.smithi059.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:43:02.068 INFO:tasks.workunit.client.0.smithi059.stdout: "summary": { 2020-03-11T22:43:02.068 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "1 pool(s) do not have an application enabled", 2020-03-11T22:43:02.068 INFO:tasks.workunit.client.0.smithi059.stdout: "count": 1 2020-03-11T22:43:02.068 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.069 INFO:tasks.workunit.client.0.smithi059.stdout: "detail": [ 2020-03-11T22:43:02.069 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.069 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "application not enabled on pool 'test'" 2020-03-11T22:43:02.069 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.069 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.069 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2020-03-11T22:43:02.069 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:02.069 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:43:02.070 INFO:tasks.workunit.client.0.smithi059.stdout: "muted": false 2020-03-11T22:43:02.070 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:02.070 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.070 INFO:tasks.workunit.client.0.smithi059.stdout: "mutes": [] 2020-03-11T22:43:02.070 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.070 INFO:tasks.workunit.client.0.smithi059.stdout: "monmap_first_committed": 1, 2020-03-11T22:43:02.070 INFO:tasks.workunit.client.0.smithi059.stdout: "monmap_last_committed": 1, 2020-03-11T22:43:02.071 INFO:tasks.workunit.client.0.smithi059.stdout: "monmap": { 2020-03-11T22:43:02.071 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": 1, 2020-03-11T22:43:02.071 INFO:tasks.workunit.client.0.smithi059.stdout: "fsid": "cb7f2e42-1934-4145-a3c3-29bdd93f8adb", 2020-03-11T22:43:02.071 INFO:tasks.workunit.client.0.smithi059.stdout: "modified": "2020-03-11T22:41:10.071517Z", 2020-03-11T22:43:02.071 INFO:tasks.workunit.client.0.smithi059.stdout: "created": "2020-03-11T22:41:10.071517Z", 2020-03-11T22:43:02.071 INFO:tasks.workunit.client.0.smithi059.stdout: "min_mon_release": 15, 2020-03-11T22:43:02.071 INFO:tasks.workunit.client.0.smithi059.stdout: "min_mon_release_name": "octopus", 2020-03-11T22:43:02.071 INFO:tasks.workunit.client.0.smithi059.stdout: "features": { 2020-03-11T22:43:02.072 INFO:tasks.workunit.client.0.smithi059.stdout: "persistent": [ 2020-03-11T22:43:02.072 INFO:tasks.workunit.client.0.smithi059.stdout: "kraken", 2020-03-11T22:43:02.072 INFO:tasks.workunit.client.0.smithi059.stdout: "luminous", 2020-03-11T22:43:02.072 INFO:tasks.workunit.client.0.smithi059.stdout: "mimic", 2020-03-11T22:43:02.072 INFO:tasks.workunit.client.0.smithi059.stdout: "osdmap-prune", 2020-03-11T22:43:02.072 INFO:tasks.workunit.client.0.smithi059.stdout: "nautilus", 2020-03-11T22:43:02.072 INFO:tasks.workunit.client.0.smithi059.stdout: "octopus" 2020-03-11T22:43:02.072 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:43:02.073 INFO:tasks.workunit.client.0.smithi059.stdout: "optional": [] 2020-03-11T22:43:02.073 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.073 INFO:tasks.workunit.client.0.smithi059.stdout: "mons": [ 2020-03-11T22:43:02.073 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.073 INFO:tasks.workunit.client.0.smithi059.stdout: "rank": 0, 2020-03-11T22:43:02.073 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "a", 2020-03-11T22:43:02.074 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addrs": { 2020-03-11T22:43:02.074 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:43:02.074 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.074 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:43:02.074 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:7115", 2020-03-11T22:43:02.075 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 0 2020-03-11T22:43:02.075 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:02.075 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:43:02.075 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.075 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:7115/0", 2020-03-11T22:43:02.075 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addr": "127.0.0.1:7115/0", 2020-03-11T22:43:02.076 INFO:tasks.workunit.client.0.smithi059.stdout: "priority": 0, 2020-03-11T22:43:02.076 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 0 2020-03-11T22:43:02.076 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:02.076 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:43:02.076 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.076 INFO:tasks.workunit.client.0.smithi059.stdout: "quorum": [ 2020-03-11T22:43:02.076 INFO:tasks.workunit.client.0.smithi059.stdout: 0 2020-03-11T22:43:02.077 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:43:02.077 INFO:tasks.workunit.client.0.smithi059.stdout: "osdmap": { 2020-03-11T22:43:02.077 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": 39, 2020-03-11T22:43:02.077 INFO:tasks.workunit.client.0.smithi059.stdout: "fsid": "cb7f2e42-1934-4145-a3c3-29bdd93f8adb", 2020-03-11T22:43:02.077 INFO:tasks.workunit.client.0.smithi059.stdout: "created": "2020-03-11T22:41:10.840470+0000", 2020-03-11T22:43:02.077 INFO:tasks.workunit.client.0.smithi059.stdout: "modified": "2020-03-11T22:42:21.008589+0000", 2020-03-11T22:43:02.077 INFO:tasks.workunit.client.0.smithi059.stdout: "last_up_change": "2020-03-11T22:42:19.357382+0000", 2020-03-11T22:43:02.077 INFO:tasks.workunit.client.0.smithi059.stdout: "last_in_change": "2020-03-11T22:42:20.779637+0000", 2020-03-11T22:43:02.078 INFO:tasks.workunit.client.0.smithi059.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-03-11T22:43:02.078 INFO:tasks.workunit.client.0.smithi059.stdout: "flags_num": 5800384, 2020-03-11T22:43:02.078 INFO:tasks.workunit.client.0.smithi059.stdout: "flags_set": [ 2020-03-11T22:43:02.078 INFO:tasks.workunit.client.0.smithi059.stdout: "nodown", 2020-03-11T22:43:02.078 INFO:tasks.workunit.client.0.smithi059.stdout: "noin", 2020-03-11T22:43:02.078 INFO:tasks.workunit.client.0.smithi059.stdout: "noout", 2020-03-11T22:43:02.078 INFO:tasks.workunit.client.0.smithi059.stdout: "pglog_hardlimit", 2020-03-11T22:43:02.079 INFO:tasks.workunit.client.0.smithi059.stdout: "purged_snapdirs", 2020-03-11T22:43:02.079 INFO:tasks.workunit.client.0.smithi059.stdout: "recovery_deletes", 2020-03-11T22:43:02.079 INFO:tasks.workunit.client.0.smithi059.stdout: "sortbitwise" 2020-03-11T22:43:02.079 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:43:02.079 INFO:tasks.workunit.client.0.smithi059.stdout: "crush_version": 8, 2020-03-11T22:43:02.079 INFO:tasks.workunit.client.0.smithi059.stdout: "full_ratio": 0.99000000953674316, 2020-03-11T22:43:02.079 INFO:tasks.workunit.client.0.smithi059.stdout: "backfillfull_ratio": 0.99000000953674316, 2020-03-11T22:43:02.079 INFO:tasks.workunit.client.0.smithi059.stdout: "nearfull_ratio": 0.85000002384185791, 2020-03-11T22:43:02.080 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_snapshot": "", 2020-03-11T22:43:02.080 INFO:tasks.workunit.client.0.smithi059.stdout: "pool_max": 1, 2020-03-11T22:43:02.080 INFO:tasks.workunit.client.0.smithi059.stdout: "max_osd": 3, 2020-03-11T22:43:02.080 INFO:tasks.workunit.client.0.smithi059.stdout: "require_min_compat_client": "jewel", 2020-03-11T22:43:02.080 INFO:tasks.workunit.client.0.smithi059.stdout: "min_compat_client": "jewel", 2020-03-11T22:43:02.080 INFO:tasks.workunit.client.0.smithi059.stdout: "require_osd_release": "octopus", 2020-03-11T22:43:02.080 INFO:tasks.workunit.client.0.smithi059.stdout: "pools": [ 2020-03-11T22:43:02.081 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.081 INFO:tasks.workunit.client.0.smithi059.stdout: "pool": 1, 2020-03-11T22:43:02.081 INFO:tasks.workunit.client.0.smithi059.stdout: "pool_name": "test", 2020-03-11T22:43:02.081 INFO:tasks.workunit.client.0.smithi059.stdout: "create_time": "2020-03-11T22:41:33.873560+0000", 2020-03-11T22:43:02.081 INFO:tasks.workunit.client.0.smithi059.stdout: "flags": 16389, 2020-03-11T22:43:02.081 INFO:tasks.workunit.client.0.smithi059.stdout: "flags_names": "hashpspool,ec_overwrites,pool_snaps", 2020-03-11T22:43:02.081 INFO:tasks.workunit.client.0.smithi059.stdout: "type": 3, 2020-03-11T22:43:02.081 INFO:tasks.workunit.client.0.smithi059.stdout: "size": 3, 2020-03-11T22:43:02.082 INFO:tasks.workunit.client.0.smithi059.stdout: "min_size": 2, 2020-03-11T22:43:02.082 INFO:tasks.workunit.client.0.smithi059.stdout: "crush_rule": 1, 2020-03-11T22:43:02.082 INFO:tasks.workunit.client.0.smithi059.stdout: "object_hash": 2, 2020-03-11T22:43:02.082 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_autoscale_mode": "off", 2020-03-11T22:43:02.082 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_num": 1, 2020-03-11T22:43:02.082 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_placement_num": 1, 2020-03-11T22:43:02.082 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_placement_num_target": 1, 2020-03-11T22:43:02.083 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_num_target": 1, 2020-03-11T22:43:02.083 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_num_pending": 1, 2020-03-11T22:43:02.083 INFO:tasks.workunit.client.0.smithi059.stdout: "last_pg_merge_meta": { 2020-03-11T22:43:02.083 INFO:tasks.workunit.client.0.smithi059.stdout: "source_pgid": "0.0", 2020-03-11T22:43:02.083 INFO:tasks.workunit.client.0.smithi059.stdout: "ready_epoch": 0, 2020-03-11T22:43:02.083 INFO:tasks.workunit.client.0.smithi059.stdout: "last_epoch_started": 0, 2020-03-11T22:43:02.083 INFO:tasks.workunit.client.0.smithi059.stdout: "last_epoch_clean": 0, 2020-03-11T22:43:02.083 INFO:tasks.workunit.client.0.smithi059.stdout: "source_version": "0'0", 2020-03-11T22:43:02.084 INFO:tasks.workunit.client.0.smithi059.stdout: "target_version": "0'0" 2020-03-11T22:43:02.084 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.084 INFO:tasks.workunit.client.0.smithi059.stdout: "last_change": "29", 2020-03-11T22:43:02.084 INFO:tasks.workunit.client.0.smithi059.stdout: "last_force_op_resend": "0", 2020-03-11T22:43:02.084 INFO:tasks.workunit.client.0.smithi059.stdout: "last_force_op_resend_prenautilus": "0", 2020-03-11T22:43:02.084 INFO:tasks.workunit.client.0.smithi059.stdout: "last_force_op_resend_preluminous": "0", 2020-03-11T22:43:02.084 INFO:tasks.workunit.client.0.smithi059.stdout: "auid": 0, 2020-03-11T22:43:02.085 INFO:tasks.workunit.client.0.smithi059.stdout: "snap_mode": "pool", 2020-03-11T22:43:02.085 INFO:tasks.workunit.client.0.smithi059.stdout: "snap_seq": 1, 2020-03-11T22:43:02.085 INFO:tasks.workunit.client.0.smithi059.stdout: "snap_epoch": 29, 2020-03-11T22:43:02.085 INFO:tasks.workunit.client.0.smithi059.stdout: "pool_snaps": [ 2020-03-11T22:43:02.085 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.085 INFO:tasks.workunit.client.0.smithi059.stdout: "snapid": 1, 2020-03-11T22:43:02.085 INFO:tasks.workunit.client.0.smithi059.stdout: "stamp": "2020-03-11T22:42:01.471185+0000", 2020-03-11T22:43:02.085 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "snap1" 2020-03-11T22:43:02.086 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:02.086 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:43:02.086 INFO:tasks.workunit.client.0.smithi059.stdout: "removed_snaps": "[]", 2020-03-11T22:43:02.086 INFO:tasks.workunit.client.0.smithi059.stdout: "quota_max_bytes": 0, 2020-03-11T22:43:02.086 INFO:tasks.workunit.client.0.smithi059.stdout: "quota_max_objects": 0, 2020-03-11T22:43:02.086 INFO:tasks.workunit.client.0.smithi059.stdout: "tiers": [], 2020-03-11T22:43:02.086 INFO:tasks.workunit.client.0.smithi059.stdout: "tier_of": -1, 2020-03-11T22:43:02.086 INFO:tasks.workunit.client.0.smithi059.stdout: "read_tier": -1, 2020-03-11T22:43:02.087 INFO:tasks.workunit.client.0.smithi059.stdout: "write_tier": -1, 2020-03-11T22:43:02.087 INFO:tasks.workunit.client.0.smithi059.stdout: "cache_mode": "none", 2020-03-11T22:43:02.087 INFO:tasks.workunit.client.0.smithi059.stdout: "target_max_bytes": 0, 2020-03-11T22:43:02.087 INFO:tasks.workunit.client.0.smithi059.stdout: "target_max_objects": 0, 2020-03-11T22:43:02.087 INFO:tasks.workunit.client.0.smithi059.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-03-11T22:43:02.087 INFO:tasks.workunit.client.0.smithi059.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-03-11T22:43:02.087 INFO:tasks.workunit.client.0.smithi059.stdout: "cache_target_full_ratio_micro": 800000, 2020-03-11T22:43:02.088 INFO:tasks.workunit.client.0.smithi059.stdout: "cache_min_flush_age": 0, 2020-03-11T22:43:02.088 INFO:tasks.workunit.client.0.smithi059.stdout: "cache_min_evict_age": 0, 2020-03-11T22:43:02.088 INFO:tasks.workunit.client.0.smithi059.stdout: "erasure_code_profile": "myprofile", 2020-03-11T22:43:02.088 INFO:tasks.workunit.client.0.smithi059.stdout: "hit_set_params": { 2020-03-11T22:43:02.088 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "none" 2020-03-11T22:43:02.088 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.088 INFO:tasks.workunit.client.0.smithi059.stdout: "hit_set_period": 0, 2020-03-11T22:43:02.089 INFO:tasks.workunit.client.0.smithi059.stdout: "hit_set_count": 0, 2020-03-11T22:43:02.089 INFO:tasks.workunit.client.0.smithi059.stdout: "use_gmt_hitset": true, 2020-03-11T22:43:02.089 INFO:tasks.workunit.client.0.smithi059.stdout: "min_read_recency_for_promote": 0, 2020-03-11T22:43:02.089 INFO:tasks.workunit.client.0.smithi059.stdout: "min_write_recency_for_promote": 0, 2020-03-11T22:43:02.089 INFO:tasks.workunit.client.0.smithi059.stdout: "hit_set_grade_decay_rate": 0, 2020-03-11T22:43:02.089 INFO:tasks.workunit.client.0.smithi059.stdout: "hit_set_search_last_n": 0, 2020-03-11T22:43:02.090 INFO:tasks.workunit.client.0.smithi059.stdout: "grade_table": [], 2020-03-11T22:43:02.090 INFO:tasks.workunit.client.0.smithi059.stdout: "stripe_width": 8192, 2020-03-11T22:43:02.090 INFO:tasks.workunit.client.0.smithi059.stdout: "expected_num_objects": 0, 2020-03-11T22:43:02.090 INFO:tasks.workunit.client.0.smithi059.stdout: "fast_read": false, 2020-03-11T22:43:02.090 INFO:tasks.workunit.client.0.smithi059.stdout: "options": {}, 2020-03-11T22:43:02.090 INFO:tasks.workunit.client.0.smithi059.stdout: "application_metadata": {} 2020-03-11T22:43:02.090 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:02.090 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:43:02.091 INFO:tasks.workunit.client.0.smithi059.stdout: "osds": [ 2020-03-11T22:43:02.091 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.091 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 0, 2020-03-11T22:43:02.091 INFO:tasks.workunit.client.0.smithi059.stdout: "uuid": "af7f49a6-1c6c-446f-802f-d448ee247966", 2020-03-11T22:43:02.091 INFO:tasks.workunit.client.0.smithi059.stdout: "up": 1, 2020-03-11T22:43:02.091 INFO:tasks.workunit.client.0.smithi059.stdout: "in": 1, 2020-03-11T22:43:02.091 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 1, 2020-03-11T22:43:02.091 INFO:tasks.workunit.client.0.smithi059.stdout: "primary_affinity": 1, 2020-03-11T22:43:02.092 INFO:tasks.workunit.client.0.smithi059.stdout: "last_clean_begin": 5, 2020-03-11T22:43:02.092 INFO:tasks.workunit.client.0.smithi059.stdout: "last_clean_end": 28, 2020-03-11T22:43:02.092 INFO:tasks.workunit.client.0.smithi059.stdout: "up_from": 35, 2020-03-11T22:43:02.092 INFO:tasks.workunit.client.0.smithi059.stdout: "up_thru": 38, 2020-03-11T22:43:02.092 INFO:tasks.workunit.client.0.smithi059.stdout: "down_at": 30, 2020-03-11T22:43:02.092 INFO:tasks.workunit.client.0.smithi059.stdout: "lost_at": 0, 2020-03-11T22:43:02.092 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addrs": { 2020-03-11T22:43:02.093 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:43:02.093 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.093 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:43:02.093 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6802", 2020-03-11T22:43:02.093 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 49144 2020-03-11T22:43:02.093 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.093 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.093 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:43:02.094 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6803", 2020-03-11T22:43:02.094 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 49144 2020-03-11T22:43:02.094 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:02.094 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:43:02.094 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.095 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_addrs": { 2020-03-11T22:43:02.095 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:43:02.095 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.095 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:43:02.095 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6804", 2020-03-11T22:43:02.095 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 49144 2020-03-11T22:43:02.095 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.095 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.096 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:43:02.096 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6805", 2020-03-11T22:43:02.096 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 49144 2020-03-11T22:43:02.096 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:02.096 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:43:02.096 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.096 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_back_addrs": { 2020-03-11T22:43:02.096 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:43:02.097 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.097 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:43:02.097 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6808", 2020-03-11T22:43:02.097 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 49144 2020-03-11T22:43:02.097 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.097 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.097 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:43:02.098 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6809", 2020-03-11T22:43:02.098 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 49144 2020-03-11T22:43:02.098 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:02.098 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:43:02.098 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.098 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_front_addrs": { 2020-03-11T22:43:02.098 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:43:02.098 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.099 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:43:02.099 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6806", 2020-03-11T22:43:02.099 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 49144 2020-03-11T22:43:02.099 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.099 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.099 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:43:02.099 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6807", 2020-03-11T22:43:02.100 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 49144 2020-03-11T22:43:02.100 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:02.100 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:43:02.100 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.100 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addr": "127.0.0.1:6803/49144", 2020-03-11T22:43:02.100 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_addr": "127.0.0.1:6805/49144", 2020-03-11T22:43:02.100 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_back_addr": "127.0.0.1:6809/49144", 2020-03-11T22:43:02.100 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_front_addr": "127.0.0.1:6807/49144", 2020-03-11T22:43:02.101 INFO:tasks.workunit.client.0.smithi059.stdout: "state": [ 2020-03-11T22:43:02.101 INFO:tasks.workunit.client.0.smithi059.stdout: "exists", 2020-03-11T22:43:02.101 INFO:tasks.workunit.client.0.smithi059.stdout: "up" 2020-03-11T22:43:02.101 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:43:02.102 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.102 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.102 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 1, 2020-03-11T22:43:02.102 INFO:tasks.workunit.client.0.smithi059.stdout: "uuid": "01a46a03-2514-4118-9d87-d9feba7ccc07", 2020-03-11T22:43:02.102 INFO:tasks.workunit.client.0.smithi059.stdout: "up": 0, 2020-03-11T22:43:02.103 INFO:tasks.workunit.client.0.smithi059.stdout: "in": 1, 2020-03-11T22:43:02.103 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 1, 2020-03-11T22:43:02.103 INFO:tasks.workunit.client.0.smithi059.stdout: "primary_affinity": 1, 2020-03-11T22:43:02.103 INFO:tasks.workunit.client.0.smithi059.stdout: "last_clean_begin": 0, 2020-03-11T22:43:02.103 INFO:tasks.workunit.client.0.smithi059.stdout: "last_clean_end": 0, 2020-03-11T22:43:02.103 INFO:tasks.workunit.client.0.smithi059.stdout: "up_from": 10, 2020-03-11T22:43:02.103 INFO:tasks.workunit.client.0.smithi059.stdout: "up_thru": 24, 2020-03-11T22:43:02.103 INFO:tasks.workunit.client.0.smithi059.stdout: "down_at": 31, 2020-03-11T22:43:02.104 INFO:tasks.workunit.client.0.smithi059.stdout: "lost_at": 0, 2020-03-11T22:43:02.104 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addrs": { 2020-03-11T22:43:02.104 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:43:02.104 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.104 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:43:02.104 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:43:02.104 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 44413 2020-03-11T22:43:02.105 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.105 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.105 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:43:02.105 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:43:02.105 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 44413 2020-03-11T22:43:02.105 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:02.105 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:43:02.105 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.106 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_addrs": { 2020-03-11T22:43:02.106 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:43:02.106 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.106 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:43:02.106 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:43:02.107 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 44413 2020-03-11T22:43:02.107 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.107 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.107 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:43:02.107 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:43:02.107 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 44413 2020-03-11T22:43:02.107 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:02.108 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:43:02.108 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.108 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_back_addrs": { 2020-03-11T22:43:02.108 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:43:02.109 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.109 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:43:02.109 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:43:02.109 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 44413 2020-03-11T22:43:02.109 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.110 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.110 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:43:02.110 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:43:02.110 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 44413 2020-03-11T22:43:02.110 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:02.110 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:43:02.110 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.110 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_front_addrs": { 2020-03-11T22:43:02.111 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:43:02.111 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.111 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:43:02.111 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:43:02.111 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 44413 2020-03-11T22:43:02.111 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.111 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.112 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:43:02.112 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:43:02.112 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 44413 2020-03-11T22:43:02.112 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:02.112 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:43:02.112 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.112 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addr": "127.0.0.1:6811/44413", 2020-03-11T22:43:02.113 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_addr": "127.0.0.1:6813/44413", 2020-03-11T22:43:02.113 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_back_addr": "127.0.0.1:6817/44413", 2020-03-11T22:43:02.113 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_front_addr": "127.0.0.1:6815/44413", 2020-03-11T22:43:02.113 INFO:tasks.workunit.client.0.smithi059.stdout: "state": [ 2020-03-11T22:43:02.113 INFO:tasks.workunit.client.0.smithi059.stdout: "exists" 2020-03-11T22:43:02.113 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:43:02.113 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.114 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.114 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 2, 2020-03-11T22:43:02.114 INFO:tasks.workunit.client.0.smithi059.stdout: "uuid": "d2482f9a-6ef9-4322-9e82-b5d8b8810499", 2020-03-11T22:43:02.114 INFO:tasks.workunit.client.0.smithi059.stdout: "up": 1, 2020-03-11T22:43:02.114 INFO:tasks.workunit.client.0.smithi059.stdout: "in": 1, 2020-03-11T22:43:02.114 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 1, 2020-03-11T22:43:02.114 INFO:tasks.workunit.client.0.smithi059.stdout: "primary_affinity": 1, 2020-03-11T22:43:02.115 INFO:tasks.workunit.client.0.smithi059.stdout: "last_clean_begin": 14, 2020-03-11T22:43:02.115 INFO:tasks.workunit.client.0.smithi059.stdout: "last_clean_end": 28, 2020-03-11T22:43:02.115 INFO:tasks.workunit.client.0.smithi059.stdout: "up_from": 36, 2020-03-11T22:43:02.115 INFO:tasks.workunit.client.0.smithi059.stdout: "up_thru": 0, 2020-03-11T22:43:02.115 INFO:tasks.workunit.client.0.smithi059.stdout: "down_at": 32, 2020-03-11T22:43:02.115 INFO:tasks.workunit.client.0.smithi059.stdout: "lost_at": 0, 2020-03-11T22:43:02.115 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addrs": { 2020-03-11T22:43:02.115 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:43:02.116 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.116 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:43:02.116 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:43:02.116 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 49338 2020-03-11T22:43:02.116 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.116 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.116 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:43:02.116 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:43:02.117 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 49338 2020-03-11T22:43:02.117 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:02.117 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:43:02.117 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.117 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_addrs": { 2020-03-11T22:43:02.117 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:43:02.117 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.118 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:43:02.118 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:43:02.118 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 49338 2020-03-11T22:43:02.118 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.118 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.118 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:43:02.118 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:43:02.118 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 49338 2020-03-11T22:43:02.119 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:02.119 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:43:02.119 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.119 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_back_addrs": { 2020-03-11T22:43:02.119 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:43:02.119 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.119 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:43:02.120 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:43:02.120 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 49338 2020-03-11T22:43:02.120 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.120 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.120 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:43:02.120 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:43:02.120 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 49338 2020-03-11T22:43:02.120 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:02.121 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:43:02.121 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.121 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_front_addrs": { 2020-03-11T22:43:02.121 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:43:02.121 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.121 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:43:02.121 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:43:02.122 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 49338 2020-03-11T22:43:02.122 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.122 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.122 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:43:02.122 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:43:02.122 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 49338 2020-03-11T22:43:02.122 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:02.122 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:43:02.123 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.123 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addr": "127.0.0.1:6811/49338", 2020-03-11T22:43:02.123 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_addr": "127.0.0.1:6813/49338", 2020-03-11T22:43:02.123 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_back_addr": "127.0.0.1:6817/49338", 2020-03-11T22:43:02.123 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_front_addr": "127.0.0.1:6815/49338", 2020-03-11T22:43:02.123 INFO:tasks.workunit.client.0.smithi059.stdout: "state": [ 2020-03-11T22:43:02.123 INFO:tasks.workunit.client.0.smithi059.stdout: "exists", 2020-03-11T22:43:02.123 INFO:tasks.workunit.client.0.smithi059.stdout: "up" 2020-03-11T22:43:02.124 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:43:02.124 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:02.124 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:43:02.124 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_xinfo": [ 2020-03-11T22:43:02.124 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.124 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 0, 2020-03-11T22:43:02.124 INFO:tasks.workunit.client.0.smithi059.stdout: "down_stamp": "2020-03-11T22:42:12.391533+0000", 2020-03-11T22:43:02.125 INFO:tasks.workunit.client.0.smithi059.stdout: "laggy_probability": 0, 2020-03-11T22:43:02.125 INFO:tasks.workunit.client.0.smithi059.stdout: "laggy_interval": 0, 2020-03-11T22:43:02.125 INFO:tasks.workunit.client.0.smithi059.stdout: "features": 4540138292836696063, 2020-03-11T22:43:02.125 INFO:tasks.workunit.client.0.smithi059.stdout: "old_weight": 0, 2020-03-11T22:43:02.125 INFO:tasks.workunit.client.0.smithi059.stdout: "last_purged_snaps_scrub": "2020-03-11T22:41:17.873225+0000", 2020-03-11T22:43:02.125 INFO:tasks.workunit.client.0.smithi059.stdout: "dead_epoch": 0 2020-03-11T22:43:02.125 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.125 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.126 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 1, 2020-03-11T22:43:02.126 INFO:tasks.workunit.client.0.smithi059.stdout: "down_stamp": "2020-03-11T22:42:13.111456+0000", 2020-03-11T22:43:02.126 INFO:tasks.workunit.client.0.smithi059.stdout: "laggy_probability": 0, 2020-03-11T22:43:02.126 INFO:tasks.workunit.client.0.smithi059.stdout: "laggy_interval": 0, 2020-03-11T22:43:02.126 INFO:tasks.workunit.client.0.smithi059.stdout: "features": 4540138292836696063, 2020-03-11T22:43:02.126 INFO:tasks.workunit.client.0.smithi059.stdout: "old_weight": 0, 2020-03-11T22:43:02.126 INFO:tasks.workunit.client.0.smithi059.stdout: "last_purged_snaps_scrub": "2020-03-11T22:41:24.468813+0000", 2020-03-11T22:43:02.127 INFO:tasks.workunit.client.0.smithi059.stdout: "dead_epoch": 0 2020-03-11T22:43:02.127 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.127 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.127 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 2, 2020-03-11T22:43:02.127 INFO:tasks.workunit.client.0.smithi059.stdout: "down_stamp": "2020-03-11T22:42:13.673781+0000", 2020-03-11T22:43:02.127 INFO:tasks.workunit.client.0.smithi059.stdout: "laggy_probability": 0, 2020-03-11T22:43:02.127 INFO:tasks.workunit.client.0.smithi059.stdout: "laggy_interval": 0, 2020-03-11T22:43:02.128 INFO:tasks.workunit.client.0.smithi059.stdout: "features": 4540138292836696063, 2020-03-11T22:43:02.128 INFO:tasks.workunit.client.0.smithi059.stdout: "old_weight": 0, 2020-03-11T22:43:02.128 INFO:tasks.workunit.client.0.smithi059.stdout: "last_purged_snaps_scrub": "2020-03-11T22:41:30.922789+0000", 2020-03-11T22:43:02.128 INFO:tasks.workunit.client.0.smithi059.stdout: "dead_epoch": 0 2020-03-11T22:43:02.128 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:02.128 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:43:02.128 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_upmap": [], 2020-03-11T22:43:02.128 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_upmap_items": [], 2020-03-11T22:43:02.129 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_temp": [], 2020-03-11T22:43:02.129 INFO:tasks.workunit.client.0.smithi059.stdout: "primary_temp": [], 2020-03-11T22:43:02.129 INFO:tasks.workunit.client.0.smithi059.stdout: "blacklist": {}, 2020-03-11T22:43:02.129 INFO:tasks.workunit.client.0.smithi059.stdout: "erasure_code_profiles": { 2020-03-11T22:43:02.129 INFO:tasks.workunit.client.0.smithi059.stdout: "default": { 2020-03-11T22:43:02.129 INFO:tasks.workunit.client.0.smithi059.stdout: "k": "2", 2020-03-11T22:43:02.130 INFO:tasks.workunit.client.0.smithi059.stdout: "m": "2", 2020-03-11T22:43:02.130 INFO:tasks.workunit.client.0.smithi059.stdout: "plugin": "jerasure", 2020-03-11T22:43:02.130 INFO:tasks.workunit.client.0.smithi059.stdout: "technique": "reed_sol_van" 2020-03-11T22:43:02.130 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.130 INFO:tasks.workunit.client.0.smithi059.stdout: "myprofile": { 2020-03-11T22:43:02.130 INFO:tasks.workunit.client.0.smithi059.stdout: "crush-device-class": "", 2020-03-11T22:43:02.130 INFO:tasks.workunit.client.0.smithi059.stdout: "crush-failure-domain": "osd", 2020-03-11T22:43:02.131 INFO:tasks.workunit.client.0.smithi059.stdout: "crush-root": "default", 2020-03-11T22:43:02.131 INFO:tasks.workunit.client.0.smithi059.stdout: "jerasure-per-chunk-alignment": "false", 2020-03-11T22:43:02.131 INFO:tasks.workunit.client.0.smithi059.stdout: "k": "2", 2020-03-11T22:43:02.131 INFO:tasks.workunit.client.0.smithi059.stdout: "m": "1", 2020-03-11T22:43:02.131 INFO:tasks.workunit.client.0.smithi059.stdout: "plugin": "jerasure", 2020-03-11T22:43:02.131 INFO:tasks.workunit.client.0.smithi059.stdout: "technique": "reed_sol_van", 2020-03-11T22:43:02.131 INFO:tasks.workunit.client.0.smithi059.stdout: "w": "8" 2020-03-11T22:43:02.132 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:02.132 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.132 INFO:tasks.workunit.client.0.smithi059.stdout: "removed_snaps_queue": [], 2020-03-11T22:43:02.132 INFO:tasks.workunit.client.0.smithi059.stdout: "new_removed_snaps": [], 2020-03-11T22:43:02.132 INFO:tasks.workunit.client.0.smithi059.stdout: "new_purged_snaps": [], 2020-03-11T22:43:02.132 INFO:tasks.workunit.client.0.smithi059.stdout: "crush_node_flags": {}, 2020-03-11T22:43:02.132 INFO:tasks.workunit.client.0.smithi059.stdout: "device_class_flags": {} 2020-03-11T22:43:02.132 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.133 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_metadata": [ 2020-03-11T22:43:02.133 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.133 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 0, 2020-03-11T22:43:02.133 INFO:tasks.workunit.client.0.smithi059.stdout: "arch": "x86_64", 2020-03-11T22:43:02.133 INFO:tasks.workunit.client.0.smithi059.stdout: "back_addr": "[v2:127.0.0.1:6804/49144,v1:127.0.0.1:6805/49144]", 2020-03-11T22:43:02.133 INFO:tasks.workunit.client.0.smithi059.stdout: "back_iface": "", 2020-03-11T22:43:02.133 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs": "1", 2020-03-11T22:43:02.134 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:43:02.134 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:43:02.134 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:43:02.134 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:43:02.134 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:43:02.134 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:43:02.134 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:43:02.135 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2020-03-11T22:43:02.135 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:43:02.135 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:43:02.135 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:43:02.135 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:43:02.135 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_release": "octopus", 2020-03-11T22:43:02.136 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:43:02.136 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:43:02.136 INFO:tasks.workunit.client.0.smithi059.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-03-11T22:43:02.136 INFO:tasks.workunit.client.0.smithi059.stdout: "default_device_class": "hdd", 2020-03-11T22:43:02.136 INFO:tasks.workunit.client.0.smithi059.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4J2P7", 2020-03-11T22:43:02.136 INFO:tasks.workunit.client.0.smithi059.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:43:02.136 INFO:tasks.workunit.client.0.smithi059.stdout: "devices": "sda", 2020-03-11T22:43:02.137 INFO:tasks.workunit.client.0.smithi059.stdout: "distro": "centos", 2020-03-11T22:43:02.137 INFO:tasks.workunit.client.0.smithi059.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:43:02.137 INFO:tasks.workunit.client.0.smithi059.stdout: "distro_version": "8", 2020-03-11T22:43:02.137 INFO:tasks.workunit.client.0.smithi059.stdout: "front_addr": "[v2:127.0.0.1:6802/49144,v1:127.0.0.1:6803/49144]", 2020-03-11T22:43:02.137 INFO:tasks.workunit.client.0.smithi059.stdout: "front_iface": "", 2020-03-11T22:43:02.137 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/49144,v1:127.0.0.1:6809/49144]", 2020-03-11T22:43:02.137 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/49144,v1:127.0.0.1:6807/49144]", 2020-03-11T22:43:02.137 INFO:tasks.workunit.client.0.smithi059.stdout: "hostname": "smithi059", 2020-03-11T22:43:02.138 INFO:tasks.workunit.client.0.smithi059.stdout: "journal_rotational": "1", 2020-03-11T22:43:02.138 INFO:tasks.workunit.client.0.smithi059.stdout: "kernel_description": "#1 SMP Wed Feb 5 02:00:39 UTC 2020", 2020-03-11T22:43:02.138 INFO:tasks.workunit.client.0.smithi059.stdout: "kernel_version": "4.18.0-147.5.1.el8_1.x86_64", 2020-03-11T22:43:02.138 INFO:tasks.workunit.client.0.smithi059.stdout: "mem_swap_kb": "0", 2020-03-11T22:43:02.138 INFO:tasks.workunit.client.0.smithi059.stdout: "mem_total_kb": "32662084", 2020-03-11T22:43:02.138 INFO:tasks.workunit.client.0.smithi059.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:43:02.138 INFO:tasks.workunit.client.0.smithi059.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:43:02.139 INFO:tasks.workunit.client.0.smithi059.stdout: "os": "Linux", 2020-03-11T22:43:02.139 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_data": "td/divergent-priors/0", 2020-03-11T22:43:02.139 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:43:02.139 INFO:tasks.workunit.client.0.smithi059.stdout: "rotational": "1" 2020-03-11T22:43:02.139 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.139 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.139 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 1, 2020-03-11T22:43:02.140 INFO:tasks.workunit.client.0.smithi059.stdout: "arch": "x86_64", 2020-03-11T22:43:02.140 INFO:tasks.workunit.client.0.smithi059.stdout: "back_addr": "[v2:127.0.0.1:6812/44413,v1:127.0.0.1:6813/44413]", 2020-03-11T22:43:02.140 INFO:tasks.workunit.client.0.smithi059.stdout: "back_iface": "", 2020-03-11T22:43:02.140 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs": "1", 2020-03-11T22:43:02.140 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:43:02.140 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:43:02.141 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:43:02.141 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:43:02.141 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:43:02.141 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:43:02.141 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:43:02.141 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2020-03-11T22:43:02.141 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:43:02.142 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:43:02.142 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:43:02.142 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:43:02.142 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_release": "octopus", 2020-03-11T22:43:02.142 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:43:02.143 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:43:02.143 INFO:tasks.workunit.client.0.smithi059.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-03-11T22:43:02.143 INFO:tasks.workunit.client.0.smithi059.stdout: "default_device_class": "hdd", 2020-03-11T22:43:02.143 INFO:tasks.workunit.client.0.smithi059.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4J2P7", 2020-03-11T22:43:02.143 INFO:tasks.workunit.client.0.smithi059.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:43:02.143 INFO:tasks.workunit.client.0.smithi059.stdout: "devices": "sda", 2020-03-11T22:43:02.143 INFO:tasks.workunit.client.0.smithi059.stdout: "distro": "centos", 2020-03-11T22:43:02.144 INFO:tasks.workunit.client.0.smithi059.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:43:02.144 INFO:tasks.workunit.client.0.smithi059.stdout: "distro_version": "8", 2020-03-11T22:43:02.144 INFO:tasks.workunit.client.0.smithi059.stdout: "front_addr": "[v2:127.0.0.1:6810/44413,v1:127.0.0.1:6811/44413]", 2020-03-11T22:43:02.144 INFO:tasks.workunit.client.0.smithi059.stdout: "front_iface": "", 2020-03-11T22:43:02.144 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/44413,v1:127.0.0.1:6817/44413]", 2020-03-11T22:43:02.144 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/44413,v1:127.0.0.1:6815/44413]", 2020-03-11T22:43:02.145 INFO:tasks.workunit.client.0.smithi059.stdout: "hostname": "smithi059", 2020-03-11T22:43:02.145 INFO:tasks.workunit.client.0.smithi059.stdout: "journal_rotational": "1", 2020-03-11T22:43:02.145 INFO:tasks.workunit.client.0.smithi059.stdout: "kernel_description": "#1 SMP Wed Feb 5 02:00:39 UTC 2020", 2020-03-11T22:43:02.145 INFO:tasks.workunit.client.0.smithi059.stdout: "kernel_version": "4.18.0-147.5.1.el8_1.x86_64", 2020-03-11T22:43:02.146 INFO:tasks.workunit.client.0.smithi059.stdout: "mem_swap_kb": "0", 2020-03-11T22:43:02.146 INFO:tasks.workunit.client.0.smithi059.stdout: "mem_total_kb": "32662084", 2020-03-11T22:43:02.146 INFO:tasks.workunit.client.0.smithi059.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:43:02.146 INFO:tasks.workunit.client.0.smithi059.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:43:02.146 INFO:tasks.workunit.client.0.smithi059.stdout: "os": "Linux", 2020-03-11T22:43:02.147 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_data": "td/divergent-priors/1", 2020-03-11T22:43:02.147 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:43:02.147 INFO:tasks.workunit.client.0.smithi059.stdout: "rotational": "1" 2020-03-11T22:43:02.147 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.148 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.148 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 2, 2020-03-11T22:43:02.148 INFO:tasks.workunit.client.0.smithi059.stdout: "arch": "x86_64", 2020-03-11T22:43:02.148 INFO:tasks.workunit.client.0.smithi059.stdout: "back_addr": "[v2:127.0.0.1:6812/49338,v1:127.0.0.1:6813/49338]", 2020-03-11T22:43:02.148 INFO:tasks.workunit.client.0.smithi059.stdout: "back_iface": "", 2020-03-11T22:43:02.148 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs": "1", 2020-03-11T22:43:02.148 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:43:02.149 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:43:02.149 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:43:02.149 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:43:02.149 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:43:02.149 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:43:02.149 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:43:02.149 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2020-03-11T22:43:02.150 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:43:02.150 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:43:02.150 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:43:02.150 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:43:02.150 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_release": "octopus", 2020-03-11T22:43:02.150 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:43:02.150 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:43:02.151 INFO:tasks.workunit.client.0.smithi059.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-03-11T22:43:02.151 INFO:tasks.workunit.client.0.smithi059.stdout: "default_device_class": "hdd", 2020-03-11T22:43:02.151 INFO:tasks.workunit.client.0.smithi059.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4J2P7", 2020-03-11T22:43:02.151 INFO:tasks.workunit.client.0.smithi059.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:43:02.151 INFO:tasks.workunit.client.0.smithi059.stdout: "devices": "sda", 2020-03-11T22:43:02.151 INFO:tasks.workunit.client.0.smithi059.stdout: "distro": "centos", 2020-03-11T22:43:02.152 INFO:tasks.workunit.client.0.smithi059.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:43:02.152 INFO:tasks.workunit.client.0.smithi059.stdout: "distro_version": "8", 2020-03-11T22:43:02.152 INFO:tasks.workunit.client.0.smithi059.stdout: "front_addr": "[v2:127.0.0.1:6810/49338,v1:127.0.0.1:6811/49338]", 2020-03-11T22:43:02.152 INFO:tasks.workunit.client.0.smithi059.stdout: "front_iface": "", 2020-03-11T22:43:02.152 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/49338,v1:127.0.0.1:6817/49338]", 2020-03-11T22:43:02.152 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/49338,v1:127.0.0.1:6815/49338]", 2020-03-11T22:43:02.152 INFO:tasks.workunit.client.0.smithi059.stdout: "hostname": "smithi059", 2020-03-11T22:43:02.153 INFO:tasks.workunit.client.0.smithi059.stdout: "journal_rotational": "1", 2020-03-11T22:43:02.153 INFO:tasks.workunit.client.0.smithi059.stdout: "kernel_description": "#1 SMP Wed Feb 5 02:00:39 UTC 2020", 2020-03-11T22:43:02.153 INFO:tasks.workunit.client.0.smithi059.stdout: "kernel_version": "4.18.0-147.5.1.el8_1.x86_64", 2020-03-11T22:43:02.153 INFO:tasks.workunit.client.0.smithi059.stdout: "mem_swap_kb": "0", 2020-03-11T22:43:02.154 INFO:tasks.workunit.client.0.smithi059.stdout: "mem_total_kb": "32662084", 2020-03-11T22:43:02.154 INFO:tasks.workunit.client.0.smithi059.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:43:02.154 INFO:tasks.workunit.client.0.smithi059.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:43:02.154 INFO:tasks.workunit.client.0.smithi059.stdout: "os": "Linux", 2020-03-11T22:43:02.154 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_data": "td/divergent-priors/2", 2020-03-11T22:43:02.154 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:43:02.155 INFO:tasks.workunit.client.0.smithi059.stdout: "rotational": "1" 2020-03-11T22:43:02.155 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:02.155 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:43:02.155 INFO:tasks.workunit.client.0.smithi059.stdout: "osdmap_first_committed": 1, 2020-03-11T22:43:02.155 INFO:tasks.workunit.client.0.smithi059.stdout: "osdmap_last_committed": 39, 2020-03-11T22:43:02.155 INFO:tasks.workunit.client.0.smithi059.stdout: "crushmap": { 2020-03-11T22:43:02.155 INFO:tasks.workunit.client.0.smithi059.stdout: "devices": [ 2020-03-11T22:43:02.155 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.156 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 0, 2020-03-11T22:43:02.156 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "osd.0", 2020-03-11T22:43:02.156 INFO:tasks.workunit.client.0.smithi059.stdout: "class": "hdd" 2020-03-11T22:43:02.156 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.156 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.156 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 1, 2020-03-11T22:43:02.156 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "osd.1", 2020-03-11T22:43:02.157 INFO:tasks.workunit.client.0.smithi059.stdout: "class": "hdd" 2020-03-11T22:43:02.157 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.157 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.157 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 2, 2020-03-11T22:43:02.157 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "osd.2", 2020-03-11T22:43:02.157 INFO:tasks.workunit.client.0.smithi059.stdout: "class": "hdd" 2020-03-11T22:43:02.158 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:02.158 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:43:02.158 INFO:tasks.workunit.client.0.smithi059.stdout: "types": [ 2020-03-11T22:43:02.158 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.158 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 0, 2020-03-11T22:43:02.158 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "osd" 2020-03-11T22:43:02.158 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.159 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.159 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 1, 2020-03-11T22:43:02.159 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "host" 2020-03-11T22:43:02.159 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.159 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.159 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 2, 2020-03-11T22:43:02.159 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "chassis" 2020-03-11T22:43:02.159 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.160 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.160 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 3, 2020-03-11T22:43:02.160 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "rack" 2020-03-11T22:43:02.160 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.160 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.160 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 4, 2020-03-11T22:43:02.160 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "row" 2020-03-11T22:43:02.161 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.161 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.161 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 5, 2020-03-11T22:43:02.161 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "pdu" 2020-03-11T22:43:02.161 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.161 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.161 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 6, 2020-03-11T22:43:02.161 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "pod" 2020-03-11T22:43:02.162 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.162 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.162 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 7, 2020-03-11T22:43:02.162 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "room" 2020-03-11T22:43:02.162 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.162 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.162 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 8, 2020-03-11T22:43:02.162 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "datacenter" 2020-03-11T22:43:02.163 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.163 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.163 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 9, 2020-03-11T22:43:02.163 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "zone" 2020-03-11T22:43:02.163 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.163 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.163 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 10, 2020-03-11T22:43:02.164 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "region" 2020-03-11T22:43:02.164 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.164 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.164 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 11, 2020-03-11T22:43:02.164 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "root" 2020-03-11T22:43:02.164 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:02.164 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:43:02.164 INFO:tasks.workunit.client.0.smithi059.stdout: "buckets": [ 2020-03-11T22:43:02.165 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.165 INFO:tasks.workunit.client.0.smithi059.stdout: "id": -1, 2020-03-11T22:43:02.165 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "default", 2020-03-11T22:43:02.165 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 11, 2020-03-11T22:43:02.165 INFO:tasks.workunit.client.0.smithi059.stdout: "type_name": "root", 2020-03-11T22:43:02.165 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 19206, 2020-03-11T22:43:02.165 INFO:tasks.workunit.client.0.smithi059.stdout: "alg": "straw2", 2020-03-11T22:43:02.166 INFO:tasks.workunit.client.0.smithi059.stdout: "hash": "rjenkins1", 2020-03-11T22:43:02.166 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2020-03-11T22:43:02.166 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.166 INFO:tasks.workunit.client.0.smithi059.stdout: "id": -3, 2020-03-11T22:43:02.166 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 19206, 2020-03-11T22:43:02.166 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 0 2020-03-11T22:43:02.166 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:02.166 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:43:02.167 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.167 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.167 INFO:tasks.workunit.client.0.smithi059.stdout: "id": -2, 2020-03-11T22:43:02.167 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "default~hdd", 2020-03-11T22:43:02.167 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 11, 2020-03-11T22:43:02.167 INFO:tasks.workunit.client.0.smithi059.stdout: "type_name": "root", 2020-03-11T22:43:02.167 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 19206, 2020-03-11T22:43:02.168 INFO:tasks.workunit.client.0.smithi059.stdout: "alg": "straw2", 2020-03-11T22:43:02.168 INFO:tasks.workunit.client.0.smithi059.stdout: "hash": "rjenkins1", 2020-03-11T22:43:02.168 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2020-03-11T22:43:02.168 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.168 INFO:tasks.workunit.client.0.smithi059.stdout: "id": -4, 2020-03-11T22:43:02.168 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 19206, 2020-03-11T22:43:02.168 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 0 2020-03-11T22:43:02.168 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:02.169 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:43:02.169 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.169 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.169 INFO:tasks.workunit.client.0.smithi059.stdout: "id": -3, 2020-03-11T22:43:02.169 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "smithi059", 2020-03-11T22:43:02.169 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 1, 2020-03-11T22:43:02.169 INFO:tasks.workunit.client.0.smithi059.stdout: "type_name": "host", 2020-03-11T22:43:02.169 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 19206, 2020-03-11T22:43:02.170 INFO:tasks.workunit.client.0.smithi059.stdout: "alg": "straw2", 2020-03-11T22:43:02.170 INFO:tasks.workunit.client.0.smithi059.stdout: "hash": "rjenkins1", 2020-03-11T22:43:02.170 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2020-03-11T22:43:02.170 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.170 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 0, 2020-03-11T22:43:02.170 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 6402, 2020-03-11T22:43:02.170 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 0 2020-03-11T22:43:02.171 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.171 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.171 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 1, 2020-03-11T22:43:02.171 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 6402, 2020-03-11T22:43:02.171 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 1 2020-03-11T22:43:02.171 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.171 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.171 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 2, 2020-03-11T22:43:02.172 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 6402, 2020-03-11T22:43:02.172 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 2 2020-03-11T22:43:02.172 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:02.172 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:43:02.172 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.172 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.172 INFO:tasks.workunit.client.0.smithi059.stdout: "id": -4, 2020-03-11T22:43:02.173 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "smithi059~hdd", 2020-03-11T22:43:02.173 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 1, 2020-03-11T22:43:02.173 INFO:tasks.workunit.client.0.smithi059.stdout: "type_name": "host", 2020-03-11T22:43:02.173 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 19206, 2020-03-11T22:43:02.173 INFO:tasks.workunit.client.0.smithi059.stdout: "alg": "straw2", 2020-03-11T22:43:02.173 INFO:tasks.workunit.client.0.smithi059.stdout: "hash": "rjenkins1", 2020-03-11T22:43:02.173 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2020-03-11T22:43:02.173 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.174 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 0, 2020-03-11T22:43:02.174 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 6402, 2020-03-11T22:43:02.174 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 0 2020-03-11T22:43:02.174 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.174 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.174 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 1, 2020-03-11T22:43:02.174 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 6402, 2020-03-11T22:43:02.175 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 1 2020-03-11T22:43:02.175 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.175 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.175 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 2, 2020-03-11T22:43:02.175 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 6402, 2020-03-11T22:43:02.175 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 2 2020-03-11T22:43:02.175 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:02.175 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:43:02.176 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:02.176 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:43:02.176 INFO:tasks.workunit.client.0.smithi059.stdout: "rules": [ 2020-03-11T22:43:02.176 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.176 INFO:tasks.workunit.client.0.smithi059.stdout: "rule_id": 0, 2020-03-11T22:43:02.176 INFO:tasks.workunit.client.0.smithi059.stdout: "rule_name": "replicated_rule", 2020-03-11T22:43:02.176 INFO:tasks.workunit.client.0.smithi059.stdout: "ruleset": 0, 2020-03-11T22:43:02.177 INFO:tasks.workunit.client.0.smithi059.stdout: "type": 1, 2020-03-11T22:43:02.177 INFO:tasks.workunit.client.0.smithi059.stdout: "min_size": 1, 2020-03-11T22:43:02.177 INFO:tasks.workunit.client.0.smithi059.stdout: "max_size": 10, 2020-03-11T22:43:02.177 INFO:tasks.workunit.client.0.smithi059.stdout: "steps": [ 2020-03-11T22:43:02.177 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.177 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "take", 2020-03-11T22:43:02.177 INFO:tasks.workunit.client.0.smithi059.stdout: "item": -1, 2020-03-11T22:43:02.177 INFO:tasks.workunit.client.0.smithi059.stdout: "item_name": "default" 2020-03-11T22:43:02.178 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.178 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.178 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "choose_firstn", 2020-03-11T22:43:02.178 INFO:tasks.workunit.client.0.smithi059.stdout: "num": 0, 2020-03-11T22:43:02.178 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "osd" 2020-03-11T22:43:02.178 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.178 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.179 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "emit" 2020-03-11T22:43:02.179 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:02.179 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:43:02.179 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.179 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.179 INFO:tasks.workunit.client.0.smithi059.stdout: "rule_id": 1, 2020-03-11T22:43:02.179 INFO:tasks.workunit.client.0.smithi059.stdout: "rule_name": "test", 2020-03-11T22:43:02.180 INFO:tasks.workunit.client.0.smithi059.stdout: "ruleset": 1, 2020-03-11T22:43:02.180 INFO:tasks.workunit.client.0.smithi059.stdout: "type": 3, 2020-03-11T22:43:02.180 INFO:tasks.workunit.client.0.smithi059.stdout: "min_size": 3, 2020-03-11T22:43:02.180 INFO:tasks.workunit.client.0.smithi059.stdout: "max_size": 3, 2020-03-11T22:43:02.180 INFO:tasks.workunit.client.0.smithi059.stdout: "steps": [ 2020-03-11T22:43:02.180 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.180 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "set_chooseleaf_tries", 2020-03-11T22:43:02.180 INFO:tasks.workunit.client.0.smithi059.stdout: "num": 5 2020-03-11T22:43:02.181 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.181 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.181 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "set_choose_tries", 2020-03-11T22:43:02.181 INFO:tasks.workunit.client.0.smithi059.stdout: "num": 100 2020-03-11T22:43:02.181 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.181 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.181 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "take", 2020-03-11T22:43:02.182 INFO:tasks.workunit.client.0.smithi059.stdout: "item": -1, 2020-03-11T22:43:02.182 INFO:tasks.workunit.client.0.smithi059.stdout: "item_name": "default" 2020-03-11T22:43:02.182 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.182 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.182 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "choose_indep", 2020-03-11T22:43:02.182 INFO:tasks.workunit.client.0.smithi059.stdout: "num": 0, 2020-03-11T22:43:02.182 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "osd" 2020-03-11T22:43:02.182 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.183 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.183 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "emit" 2020-03-11T22:43:02.183 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:02.183 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:43:02.183 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:02.183 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:43:02.183 INFO:tasks.workunit.client.0.smithi059.stdout: "tunables": { 2020-03-11T22:43:02.184 INFO:tasks.workunit.client.0.smithi059.stdout: "choose_local_tries": 0, 2020-03-11T22:43:02.184 INFO:tasks.workunit.client.0.smithi059.stdout: "choose_local_fallback_tries": 0, 2020-03-11T22:43:02.184 INFO:tasks.workunit.client.0.smithi059.stdout: "choose_total_tries": 50, 2020-03-11T22:43:02.184 INFO:tasks.workunit.client.0.smithi059.stdout: "chooseleaf_descend_once": 1, 2020-03-11T22:43:02.184 INFO:tasks.workunit.client.0.smithi059.stdout: "chooseleaf_vary_r": 1, 2020-03-11T22:43:02.184 INFO:tasks.workunit.client.0.smithi059.stdout: "chooseleaf_stable": 1, 2020-03-11T22:43:02.184 INFO:tasks.workunit.client.0.smithi059.stdout: "straw_calc_version": 1, 2020-03-11T22:43:02.184 INFO:tasks.workunit.client.0.smithi059.stdout: "allowed_bucket_algs": 54, 2020-03-11T22:43:02.185 INFO:tasks.workunit.client.0.smithi059.stdout: "profile": "jewel", 2020-03-11T22:43:02.185 INFO:tasks.workunit.client.0.smithi059.stdout: "optimal_tunables": 1, 2020-03-11T22:43:02.185 INFO:tasks.workunit.client.0.smithi059.stdout: "legacy_tunables": 0, 2020-03-11T22:43:02.185 INFO:tasks.workunit.client.0.smithi059.stdout: "minimum_required_version": "jewel", 2020-03-11T22:43:02.185 INFO:tasks.workunit.client.0.smithi059.stdout: "require_feature_tunables": 1, 2020-03-11T22:43:02.185 INFO:tasks.workunit.client.0.smithi059.stdout: "require_feature_tunables2": 1, 2020-03-11T22:43:02.185 INFO:tasks.workunit.client.0.smithi059.stdout: "has_v2_rules": 1, 2020-03-11T22:43:02.186 INFO:tasks.workunit.client.0.smithi059.stdout: "require_feature_tunables3": 1, 2020-03-11T22:43:02.186 INFO:tasks.workunit.client.0.smithi059.stdout: "has_v3_rules": 0, 2020-03-11T22:43:02.186 INFO:tasks.workunit.client.0.smithi059.stdout: "has_v4_buckets": 1, 2020-03-11T22:43:02.186 INFO:tasks.workunit.client.0.smithi059.stdout: "require_feature_tunables5": 1, 2020-03-11T22:43:02.186 INFO:tasks.workunit.client.0.smithi059.stdout: "has_v5_rules": 0 2020-03-11T22:43:02.186 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.186 INFO:tasks.workunit.client.0.smithi059.stdout: "choose_args": {} 2020-03-11T22:43:02.186 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.187 INFO:tasks.workunit.client.0.smithi059.stdout: "fsmap": { 2020-03-11T22:43:02.187 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": 1, 2020-03-11T22:43:02.187 INFO:tasks.workunit.client.0.smithi059.stdout: "default_fscid": -1, 2020-03-11T22:43:02.187 INFO:tasks.workunit.client.0.smithi059.stdout: "compat": { 2020-03-11T22:43:02.187 INFO:tasks.workunit.client.0.smithi059.stdout: "compat": {}, 2020-03-11T22:43:02.187 INFO:tasks.workunit.client.0.smithi059.stdout: "ro_compat": {}, 2020-03-11T22:43:02.187 INFO:tasks.workunit.client.0.smithi059.stdout: "incompat": { 2020-03-11T22:43:02.188 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_1": "base v0.20", 2020-03-11T22:43:02.188 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_2": "client writeable ranges", 2020-03-11T22:43:02.188 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_3": "default file layouts on dirs", 2020-03-11T22:43:02.188 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_4": "dir inode in separate object", 2020-03-11T22:43:02.188 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_5": "mds uses versioned encoding", 2020-03-11T22:43:02.188 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_6": "dirfrag is stored in omap", 2020-03-11T22:43:02.188 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_8": "no anchor table", 2020-03-11T22:43:02.188 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_9": "file layout v2", 2020-03-11T22:43:02.189 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_10": "snaprealm v2" 2020-03-11T22:43:02.189 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:02.189 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.189 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_flags": { 2020-03-11T22:43:02.189 INFO:tasks.workunit.client.0.smithi059.stdout: "enable_multiple": false, 2020-03-11T22:43:02.189 INFO:tasks.workunit.client.0.smithi059.stdout: "ever_enabled_multiple": false 2020-03-11T22:43:02.190 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.190 INFO:tasks.workunit.client.0.smithi059.stdout: "standbys": [], 2020-03-11T22:43:02.190 INFO:tasks.workunit.client.0.smithi059.stdout: "filesystems": [] 2020-03-11T22:43:02.190 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.190 INFO:tasks.workunit.client.0.smithi059.stdout: "mdsmap_first_committed": 1, 2020-03-11T22:43:02.190 INFO:tasks.workunit.client.0.smithi059.stdout: "mdsmap_last_committed": 1, 2020-03-11T22:43:02.190 INFO:tasks.workunit.client.0.smithi059.stdout: "auth": { 2020-03-11T22:43:02.191 INFO:tasks.workunit.client.0.smithi059.stdout: "first_committed": 1, 2020-03-11T22:43:02.191 INFO:tasks.workunit.client.0.smithi059.stdout: "last_committed": 5, 2020-03-11T22:43:02.191 INFO:tasks.workunit.client.0.smithi059.stdout: "num_secrets": 3 2020-03-11T22:43:02.191 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.191 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pg": 1, 2020-03-11T22:43:02.192 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pg_active": 1, 2020-03-11T22:43:02.192 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pg_unknown": 0, 2020-03-11T22:43:02.192 INFO:tasks.workunit.client.0.smithi059.stdout: "num_osd": 3, 2020-03-11T22:43:02.192 INFO:tasks.workunit.client.0.smithi059.stdout: "pool_sum": { 2020-03-11T22:43:02.192 INFO:tasks.workunit.client.0.smithi059.stdout: "stat_sum": { 2020-03-11T22:43:02.193 INFO:tasks.workunit.client.0.smithi059.stdout: "num_bytes": 48430, 2020-03-11T22:43:02.193 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects": 101, 2020-03-11T22:43:02.193 INFO:tasks.workunit.client.0.smithi059.stdout: "num_object_clones": 1, 2020-03-11T22:43:02.193 INFO:tasks.workunit.client.0.smithi059.stdout: "num_object_copies": 303, 2020-03-11T22:43:02.193 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:43:02.193 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_missing": 0, 2020-03-11T22:43:02.193 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_degraded": 101, 2020-03-11T22:43:02.194 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_misplaced": 0, 2020-03-11T22:43:02.194 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_unfound": 0, 2020-03-11T22:43:02.194 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_dirty": 101, 2020-03-11T22:43:02.194 INFO:tasks.workunit.client.0.smithi059.stdout: "num_whiteouts": 0, 2020-03-11T22:43:02.194 INFO:tasks.workunit.client.0.smithi059.stdout: "num_read": 0, 2020-03-11T22:43:02.194 INFO:tasks.workunit.client.0.smithi059.stdout: "num_read_kb": 0, 2020-03-11T22:43:02.194 INFO:tasks.workunit.client.0.smithi059.stdout: "num_write": 101, 2020-03-11T22:43:02.194 INFO:tasks.workunit.client.0.smithi059.stdout: "num_write_kb": 101, 2020-03-11T22:43:02.195 INFO:tasks.workunit.client.0.smithi059.stdout: "num_scrub_errors": 0, 2020-03-11T22:43:02.195 INFO:tasks.workunit.client.0.smithi059.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:43:02.195 INFO:tasks.workunit.client.0.smithi059.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:43:02.195 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_recovered": 0, 2020-03-11T22:43:02.195 INFO:tasks.workunit.client.0.smithi059.stdout: "num_bytes_recovered": 0, 2020-03-11T22:43:02.195 INFO:tasks.workunit.client.0.smithi059.stdout: "num_keys_recovered": 0, 2020-03-11T22:43:02.195 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_omap": 0, 2020-03-11T22:43:02.196 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:43:02.196 INFO:tasks.workunit.client.0.smithi059.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:43:02.196 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush": 0, 2020-03-11T22:43:02.196 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush_kb": 0, 2020-03-11T22:43:02.196 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict": 0, 2020-03-11T22:43:02.196 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict_kb": 0, 2020-03-11T22:43:02.196 INFO:tasks.workunit.client.0.smithi059.stdout: "num_promote": 0, 2020-03-11T22:43:02.197 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush_mode_high": 0, 2020-03-11T22:43:02.197 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush_mode_low": 0, 2020-03-11T22:43:02.197 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict_mode_some": 0, 2020-03-11T22:43:02.197 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict_mode_full": 0, 2020-03-11T22:43:02.197 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_pinned": 0, 2020-03-11T22:43:02.197 INFO:tasks.workunit.client.0.smithi059.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:43:02.197 INFO:tasks.workunit.client.0.smithi059.stdout: "num_large_omap_objects": 0, 2020-03-11T22:43:02.197 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_manifest": 0, 2020-03-11T22:43:02.198 INFO:tasks.workunit.client.0.smithi059.stdout: "num_omap_bytes": 0, 2020-03-11T22:43:02.198 INFO:tasks.workunit.client.0.smithi059.stdout: "num_omap_keys": 0, 2020-03-11T22:43:02.198 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_repaired": 0 2020-03-11T22:43:02.198 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.198 INFO:tasks.workunit.client.0.smithi059.stdout: "store_stats": { 2020-03-11T22:43:02.199 INFO:tasks.workunit.client.0.smithi059.stdout: "total": 0, 2020-03-11T22:43:02.199 INFO:tasks.workunit.client.0.smithi059.stdout: "available": 0, 2020-03-11T22:43:02.199 INFO:tasks.workunit.client.0.smithi059.stdout: "internally_reserved": 0, 2020-03-11T22:43:02.199 INFO:tasks.workunit.client.0.smithi059.stdout: "allocated": 0, 2020-03-11T22:43:02.199 INFO:tasks.workunit.client.0.smithi059.stdout: "data_stored": 0, 2020-03-11T22:43:02.199 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed": 0, 2020-03-11T22:43:02.199 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_allocated": 0, 2020-03-11T22:43:02.200 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_original": 0, 2020-03-11T22:43:02.200 INFO:tasks.workunit.client.0.smithi059.stdout: "omap_allocated": 0, 2020-03-11T22:43:02.200 INFO:tasks.workunit.client.0.smithi059.stdout: "internal_metadata": 0 2020-03-11T22:43:02.200 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.200 INFO:tasks.workunit.client.0.smithi059.stdout: "log_size": 18, 2020-03-11T22:43:02.201 INFO:tasks.workunit.client.0.smithi059.stdout: "ondisk_log_size": 18, 2020-03-11T22:43:02.201 INFO:tasks.workunit.client.0.smithi059.stdout: "up": 3, 2020-03-11T22:43:02.201 INFO:tasks.workunit.client.0.smithi059.stdout: "acting": 3, 2020-03-11T22:43:02.201 INFO:tasks.workunit.client.0.smithi059.stdout: "num_store_stats": 0 2020-03-11T22:43:02.201 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.202 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_sum": { 2020-03-11T22:43:02.202 INFO:tasks.workunit.client.0.smithi059.stdout: "up_from": 0, 2020-03-11T22:43:02.202 INFO:tasks.workunit.client.0.smithi059.stdout: "seq": 0, 2020-03-11T22:43:02.202 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pgs": 3, 2020-03-11T22:43:02.202 INFO:tasks.workunit.client.0.smithi059.stdout: "num_osds": 3, 2020-03-11T22:43:02.202 INFO:tasks.workunit.client.0.smithi059.stdout: "num_per_pool_osds": 3, 2020-03-11T22:43:02.202 INFO:tasks.workunit.client.0.smithi059.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:43:02.202 INFO:tasks.workunit.client.0.smithi059.stdout: "kb": 314572800, 2020-03-11T22:43:02.203 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used": 3179520, 2020-03-11T22:43:02.203 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used_data": 33472, 2020-03-11T22:43:02.203 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used_omap": 11, 2020-03-11T22:43:02.203 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used_meta": 3145716, 2020-03-11T22:43:02.203 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_avail": 311393280, 2020-03-11T22:43:02.203 INFO:tasks.workunit.client.0.smithi059.stdout: "statfs": { 2020-03-11T22:43:02.203 INFO:tasks.workunit.client.0.smithi059.stdout: "total": 322122547200, 2020-03-11T22:43:02.204 INFO:tasks.workunit.client.0.smithi059.stdout: "available": 318866718720, 2020-03-11T22:43:02.204 INFO:tasks.workunit.client.0.smithi059.stdout: "internally_reserved": 3221225472, 2020-03-11T22:43:02.204 INFO:tasks.workunit.client.0.smithi059.stdout: "allocated": 34275328, 2020-03-11T22:43:02.204 INFO:tasks.workunit.client.0.smithi059.stdout: "data_stored": 1518003, 2020-03-11T22:43:02.204 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed": 0, 2020-03-11T22:43:02.204 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_allocated": 0, 2020-03-11T22:43:02.204 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_original": 0, 2020-03-11T22:43:02.204 INFO:tasks.workunit.client.0.smithi059.stdout: "omap_allocated": 11560, 2020-03-11T22:43:02.205 INFO:tasks.workunit.client.0.smithi059.stdout: "internal_metadata": 3221213912 2020-03-11T22:43:02.205 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.205 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_peers": [], 2020-03-11T22:43:02.205 INFO:tasks.workunit.client.0.smithi059.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:43:02.205 INFO:tasks.workunit.client.0.smithi059.stdout: "num_snap_trimming": 0, 2020-03-11T22:43:02.205 INFO:tasks.workunit.client.0.smithi059.stdout: "num_shards_repaired": 0, 2020-03-11T22:43:02.205 INFO:tasks.workunit.client.0.smithi059.stdout: "op_queue_age_hist": { 2020-03-11T22:43:02.206 INFO:tasks.workunit.client.0.smithi059.stdout: "histogram": [], 2020-03-11T22:43:02.206 INFO:tasks.workunit.client.0.smithi059.stdout: "upper_bound": 1 2020-03-11T22:43:02.206 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.206 INFO:tasks.workunit.client.0.smithi059.stdout: "perf_stat": { 2020-03-11T22:43:02.206 INFO:tasks.workunit.client.0.smithi059.stdout: "commit_latency_ms": 0, 2020-03-11T22:43:02.206 INFO:tasks.workunit.client.0.smithi059.stdout: "apply_latency_ms": 0, 2020-03-11T22:43:02.206 INFO:tasks.workunit.client.0.smithi059.stdout: "commit_latency_ns": 0, 2020-03-11T22:43:02.206 INFO:tasks.workunit.client.0.smithi059.stdout: "apply_latency_ns": 0 2020-03-11T22:43:02.207 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.207 INFO:tasks.workunit.client.0.smithi059.stdout: "alerts": [], 2020-03-11T22:43:02.207 INFO:tasks.workunit.client.0.smithi059.stdout: "network_ping_times": [] 2020-03-11T22:43:02.207 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.207 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_sum_by_class": { 2020-03-11T22:43:02.207 INFO:tasks.workunit.client.0.smithi059.stdout: "hdd": { 2020-03-11T22:43:02.207 INFO:tasks.workunit.client.0.smithi059.stdout: "up_from": 0, 2020-03-11T22:43:02.208 INFO:tasks.workunit.client.0.smithi059.stdout: "seq": 0, 2020-03-11T22:43:02.208 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pgs": 3, 2020-03-11T22:43:02.208 INFO:tasks.workunit.client.0.smithi059.stdout: "num_osds": 3, 2020-03-11T22:43:02.208 INFO:tasks.workunit.client.0.smithi059.stdout: "num_per_pool_osds": 3, 2020-03-11T22:43:02.208 INFO:tasks.workunit.client.0.smithi059.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:43:02.208 INFO:tasks.workunit.client.0.smithi059.stdout: "kb": 314572800, 2020-03-11T22:43:02.208 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used": 3179520, 2020-03-11T22:43:02.208 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used_data": 33472, 2020-03-11T22:43:02.209 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used_omap": 11, 2020-03-11T22:43:02.209 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used_meta": 3145716, 2020-03-11T22:43:02.209 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_avail": 311393280, 2020-03-11T22:43:02.209 INFO:tasks.workunit.client.0.smithi059.stdout: "statfs": { 2020-03-11T22:43:02.209 INFO:tasks.workunit.client.0.smithi059.stdout: "total": 322122547200, 2020-03-11T22:43:02.209 INFO:tasks.workunit.client.0.smithi059.stdout: "available": 318866718720, 2020-03-11T22:43:02.209 INFO:tasks.workunit.client.0.smithi059.stdout: "internally_reserved": 3221225472, 2020-03-11T22:43:02.210 INFO:tasks.workunit.client.0.smithi059.stdout: "allocated": 34275328, 2020-03-11T22:43:02.210 INFO:tasks.workunit.client.0.smithi059.stdout: "data_stored": 1518003, 2020-03-11T22:43:02.210 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed": 0, 2020-03-11T22:43:02.210 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_allocated": 0, 2020-03-11T22:43:02.210 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_original": 0, 2020-03-11T22:43:02.210 INFO:tasks.workunit.client.0.smithi059.stdout: "omap_allocated": 11560, 2020-03-11T22:43:02.210 INFO:tasks.workunit.client.0.smithi059.stdout: "internal_metadata": 3221213912 2020-03-11T22:43:02.210 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.211 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_peers": [], 2020-03-11T22:43:02.211 INFO:tasks.workunit.client.0.smithi059.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:43:02.211 INFO:tasks.workunit.client.0.smithi059.stdout: "num_snap_trimming": 0, 2020-03-11T22:43:02.211 INFO:tasks.workunit.client.0.smithi059.stdout: "num_shards_repaired": 0, 2020-03-11T22:43:02.211 INFO:tasks.workunit.client.0.smithi059.stdout: "op_queue_age_hist": { 2020-03-11T22:43:02.211 INFO:tasks.workunit.client.0.smithi059.stdout: "histogram": [], 2020-03-11T22:43:02.211 INFO:tasks.workunit.client.0.smithi059.stdout: "upper_bound": 1 2020-03-11T22:43:02.212 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.212 INFO:tasks.workunit.client.0.smithi059.stdout: "perf_stat": { 2020-03-11T22:43:02.212 INFO:tasks.workunit.client.0.smithi059.stdout: "commit_latency_ms": 0, 2020-03-11T22:43:02.212 INFO:tasks.workunit.client.0.smithi059.stdout: "apply_latency_ms": 0, 2020-03-11T22:43:02.212 INFO:tasks.workunit.client.0.smithi059.stdout: "commit_latency_ns": 0, 2020-03-11T22:43:02.212 INFO:tasks.workunit.client.0.smithi059.stdout: "apply_latency_ns": 0 2020-03-11T22:43:02.212 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.213 INFO:tasks.workunit.client.0.smithi059.stdout: "alerts": [], 2020-03-11T22:43:02.213 INFO:tasks.workunit.client.0.smithi059.stdout: "network_ping_times": [] 2020-03-11T22:43:02.213 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:02.213 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.213 INFO:tasks.workunit.client.0.smithi059.stdout: "pool_stats": [ 2020-03-11T22:43:02.213 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.213 INFO:tasks.workunit.client.0.smithi059.stdout: "poolid": 1, 2020-03-11T22:43:02.214 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pg": 1, 2020-03-11T22:43:02.214 INFO:tasks.workunit.client.0.smithi059.stdout: "stat_sum": { 2020-03-11T22:43:02.214 INFO:tasks.workunit.client.0.smithi059.stdout: "num_bytes": 48430, 2020-03-11T22:43:02.214 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects": 101, 2020-03-11T22:43:02.214 INFO:tasks.workunit.client.0.smithi059.stdout: "num_object_clones": 1, 2020-03-11T22:43:02.214 INFO:tasks.workunit.client.0.smithi059.stdout: "num_object_copies": 303, 2020-03-11T22:43:02.214 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:43:02.215 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_missing": 0, 2020-03-11T22:43:02.215 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_degraded": 101, 2020-03-11T22:43:02.215 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_misplaced": 0, 2020-03-11T22:43:02.215 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_unfound": 0, 2020-03-11T22:43:02.215 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_dirty": 101, 2020-03-11T22:43:02.215 INFO:tasks.workunit.client.0.smithi059.stdout: "num_whiteouts": 0, 2020-03-11T22:43:02.215 INFO:tasks.workunit.client.0.smithi059.stdout: "num_read": 0, 2020-03-11T22:43:02.216 INFO:tasks.workunit.client.0.smithi059.stdout: "num_read_kb": 0, 2020-03-11T22:43:02.216 INFO:tasks.workunit.client.0.smithi059.stdout: "num_write": 101, 2020-03-11T22:43:02.216 INFO:tasks.workunit.client.0.smithi059.stdout: "num_write_kb": 101, 2020-03-11T22:43:02.216 INFO:tasks.workunit.client.0.smithi059.stdout: "num_scrub_errors": 0, 2020-03-11T22:43:02.216 INFO:tasks.workunit.client.0.smithi059.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:43:02.216 INFO:tasks.workunit.client.0.smithi059.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:43:02.216 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_recovered": 0, 2020-03-11T22:43:02.216 INFO:tasks.workunit.client.0.smithi059.stdout: "num_bytes_recovered": 0, 2020-03-11T22:43:02.217 INFO:tasks.workunit.client.0.smithi059.stdout: "num_keys_recovered": 0, 2020-03-11T22:43:02.217 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_omap": 0, 2020-03-11T22:43:02.217 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:43:02.217 INFO:tasks.workunit.client.0.smithi059.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:43:02.217 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush": 0, 2020-03-11T22:43:02.217 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush_kb": 0, 2020-03-11T22:43:02.217 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict": 0, 2020-03-11T22:43:02.218 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict_kb": 0, 2020-03-11T22:43:02.218 INFO:tasks.workunit.client.0.smithi059.stdout: "num_promote": 0, 2020-03-11T22:43:02.218 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush_mode_high": 0, 2020-03-11T22:43:02.218 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush_mode_low": 0, 2020-03-11T22:43:02.218 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict_mode_some": 0, 2020-03-11T22:43:02.218 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict_mode_full": 0, 2020-03-11T22:43:02.218 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_pinned": 0, 2020-03-11T22:43:02.218 INFO:tasks.workunit.client.0.smithi059.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:43:02.219 INFO:tasks.workunit.client.0.smithi059.stdout: "num_large_omap_objects": 0, 2020-03-11T22:43:02.219 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_manifest": 0, 2020-03-11T22:43:02.219 INFO:tasks.workunit.client.0.smithi059.stdout: "num_omap_bytes": 0, 2020-03-11T22:43:02.219 INFO:tasks.workunit.client.0.smithi059.stdout: "num_omap_keys": 0, 2020-03-11T22:43:02.219 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_repaired": 0 2020-03-11T22:43:02.219 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.219 INFO:tasks.workunit.client.0.smithi059.stdout: "store_stats": { 2020-03-11T22:43:02.220 INFO:tasks.workunit.client.0.smithi059.stdout: "total": 0, 2020-03-11T22:43:02.220 INFO:tasks.workunit.client.0.smithi059.stdout: "available": 0, 2020-03-11T22:43:02.220 INFO:tasks.workunit.client.0.smithi059.stdout: "internally_reserved": 0, 2020-03-11T22:43:02.220 INFO:tasks.workunit.client.0.smithi059.stdout: "allocated": 19857408, 2020-03-11T22:43:02.220 INFO:tasks.workunit.client.0.smithi059.stdout: "data_stored": 1241088, 2020-03-11T22:43:02.220 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed": 0, 2020-03-11T22:43:02.220 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_allocated": 0, 2020-03-11T22:43:02.220 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_original": 0, 2020-03-11T22:43:02.221 INFO:tasks.workunit.client.0.smithi059.stdout: "omap_allocated": 0, 2020-03-11T22:43:02.221 INFO:tasks.workunit.client.0.smithi059.stdout: "internal_metadata": 0 2020-03-11T22:43:02.221 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.221 INFO:tasks.workunit.client.0.smithi059.stdout: "log_size": 18, 2020-03-11T22:43:02.221 INFO:tasks.workunit.client.0.smithi059.stdout: "ondisk_log_size": 18, 2020-03-11T22:43:02.221 INFO:tasks.workunit.client.0.smithi059.stdout: "up": 3, 2020-03-11T22:43:02.221 INFO:tasks.workunit.client.0.smithi059.stdout: "acting": 3, 2020-03-11T22:43:02.222 INFO:tasks.workunit.client.0.smithi059.stdout: "num_store_stats": 3 2020-03-11T22:43:02.222 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:02.222 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:43:02.222 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_stats": [ 2020-03-11T22:43:02.222 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.222 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 0, 2020-03-11T22:43:02.222 INFO:tasks.workunit.client.0.smithi059.stdout: "seq": 150323855371 2020-03-11T22:43:02.222 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.223 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.223 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 1, 2020-03-11T22:43:02.223 INFO:tasks.workunit.client.0.smithi059.stdout: "seq": 42949672978 2020-03-11T22:43:02.223 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.223 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.223 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 2, 2020-03-11T22:43:02.223 INFO:tasks.workunit.client.0.smithi059.stdout: "seq": 154618822667 2020-03-11T22:43:02.224 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:02.224 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:43:02.224 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pg_by_state": [ 2020-03-11T22:43:02.224 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.224 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "active+undersized+degraded", 2020-03-11T22:43:02.224 INFO:tasks.workunit.client.0.smithi059.stdout: "num": 1 2020-03-11T22:43:02.224 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:02.224 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:43:02.225 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pg_by_osd": [ 2020-03-11T22:43:02.225 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.225 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 2, 2020-03-11T22:43:02.225 INFO:tasks.workunit.client.0.smithi059.stdout: "num_primary_pg": 0, 2020-03-11T22:43:02.225 INFO:tasks.workunit.client.0.smithi059.stdout: "num_acting_pg": 1, 2020-03-11T22:43:02.225 INFO:tasks.workunit.client.0.smithi059.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:43:02.225 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.225 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.226 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 2147483647, 2020-03-11T22:43:02.226 INFO:tasks.workunit.client.0.smithi059.stdout: "num_primary_pg": 0, 2020-03-11T22:43:02.226 INFO:tasks.workunit.client.0.smithi059.stdout: "num_acting_pg": 1, 2020-03-11T22:43:02.226 INFO:tasks.workunit.client.0.smithi059.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:43:02.226 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.226 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.226 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 1, 2020-03-11T22:43:02.227 INFO:tasks.workunit.client.0.smithi059.stdout: "num_primary_pg": 0, 2020-03-11T22:43:02.227 INFO:tasks.workunit.client.0.smithi059.stdout: "num_acting_pg": 0, 2020-03-11T22:43:02.227 INFO:tasks.workunit.client.0.smithi059.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:43:02.227 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.227 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.227 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 0, 2020-03-11T22:43:02.227 INFO:tasks.workunit.client.0.smithi059.stdout: "num_primary_pg": 1, 2020-03-11T22:43:02.228 INFO:tasks.workunit.client.0.smithi059.stdout: "num_acting_pg": 1, 2020-03-11T22:43:02.228 INFO:tasks.workunit.client.0.smithi059.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:43:02.228 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:02.228 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:43:02.228 INFO:tasks.workunit.client.0.smithi059.stdout: "purged_snaps": [ 2020-03-11T22:43:02.228 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:02.228 INFO:tasks.workunit.client.0.smithi059.stdout: "pool": 1, 2020-03-11T22:43:02.228 INFO:tasks.workunit.client.0.smithi059.stdout: "purged_snaps": {} 2020-03-11T22:43:02.229 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:02.229 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:43:02.229 INFO:tasks.workunit.client.0.smithi059.stdout: "servicemap": { 2020-03-11T22:43:02.229 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": 1, 2020-03-11T22:43:02.229 INFO:tasks.workunit.client.0.smithi059.stdout: "modified": "2020-03-11T22:41:10.840801+0000", 2020-03-11T22:43:02.229 INFO:tasks.workunit.client.0.smithi059.stdout: "services": {} 2020-03-11T22:43:02.229 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:02.230 INFO:tasks.workunit.client.0.smithi059.stdout: "paxos": { 2020-03-11T22:43:02.230 INFO:tasks.workunit.client.0.smithi059.stdout: "first_committed": 1, 2020-03-11T22:43:02.230 INFO:tasks.workunit.client.0.smithi059.stdout: "last_committed": 330, 2020-03-11T22:43:02.230 INFO:tasks.workunit.client.0.smithi059.stdout: "last_pn": 0, 2020-03-11T22:43:02.230 INFO:tasks.workunit.client.0.smithi059.stdout: "accepted_pn": 0 2020-03-11T22:43:02.230 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:02.231 INFO:tasks.workunit.client.0.smithi059.stdout:} 2020-03-11T22:43:02.232 INFO:tasks.workunit.client.0.smithi059.stderr:report 2375308311 2020-03-11T22:43:02.233 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: return 1 2020-03-11T22:43:02.233 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:330: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:43:02.233 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:332: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.0 2020-03-11T22:43:02.233 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:43:02.233 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:43:02.233 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:43:02.233 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:43:02.234 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:43:02.234 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:43:02.234 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:333: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 0 --op log --pgid 1.0 2020-03-11T22:43:02.234 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/divergent-priors 2020-03-11T22:43:02.234 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-03-11T22:43:02.234 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=0 2020-03-11T22:43:02.234 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-03-11T22:43:02.235 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/divergent-priors/0 2020-03-11T22:43:02.235 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/0 --op log --pgid 1.0 2020-03-11T22:43:02.747 INFO:tasks.workunit.client.0.smithi059.stderr:PG '1.0' not found 2020-03-11T22:43:03.269 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: return 1 2020-03-11T22:43:03.270 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:334: TEST_divergent_ec: activate_osd td/divergent-priors 0 2020-03-11T22:43:03.270 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:43:03.271 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:43:03.271 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=0 2020-03-11T22:43:03.271 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:43:03.271 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/0 2020-03-11T22:43:03.272 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=cb7f2e42-1934-4145-a3c3-29bdd93f8adb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:43:03.272 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:43:03.272 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:43:03.272 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:43:03.273 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-11T22:43:03.273 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-11T22:43:03.273 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:43:03.273 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:43:03.274 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:43:03.274 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:43:03.274 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:43:03.274 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:43:03.275 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:43:03.275 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:43:03.275 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:43:03.275 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17931/$cluster-$name.asok' 2020-03-11T22:43:03.276 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' 2020-03-11T22:43:03.276 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:43:03.276 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:43:03.276 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:43:03.276 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:43:03.277 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:43:03.277 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:43:03.277 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:43:03.277 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:43:03.278 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/0 2020-03-11T22:43:03.278 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.0 2020-03-11T22:43:03.278 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 0 --fsid=cb7f2e42-1934-4145-a3c3-29bdd93f8adb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:43:03.278 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/0/whoami 2020-03-11T22:43:03.279 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 0 = 0 ']' 2020-03-11T22:43:03.279 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:43:03.279 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:43:03.279 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:43:03.280 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.0 2020-03-11T22:43:03.298 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:43:03.294+0000 7f910141fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:03.299 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:43:03.296+0000 7f910141fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:03.302 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:43:03.299+0000 7f910141fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:03.648 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 0 2020-03-11T22:43:03.648 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:43:03.648 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:43:03.649 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:43:03.649 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:43:03.649 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:43:03.649 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:43:03.649 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:43:03.649 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:43:03.650 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:43:03.817 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:43:03.814+0000 7f910141fec0 -1 Falling back to public interface 2020-03-11T22:43:04.059 INFO:tasks.workunit.client.0.smithi059.stdout:osd.0 up in weight 1 up_from 35 up_thru 38 down_at 30 last_clean_interval [5,28) [v2:127.0.0.1:6802/49144,v1:127.0.0.1:6803/49144] [v2:127.0.0.1:6804/49144,v1:127.0.0.1:6805/49144] exists,up af7f49a6-1c6c-446f-802f-d448ee247966 2020-03-11T22:43:04.060 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:43:04.061 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:43:04.061 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:43:04.061 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:330: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:43:04.061 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:332: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.2 2020-03-11T22:43:04.061 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:43:04.061 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:43:04.062 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:43:04.062 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:43:04.062 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:43:04.166 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:43:04.166 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:333: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 2 --op log --pgid 1.0 2020-03-11T22:43:04.167 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/divergent-priors 2020-03-11T22:43:04.167 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-03-11T22:43:04.167 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=2 2020-03-11T22:43:04.167 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-03-11T22:43:04.167 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/divergent-priors/2 2020-03-11T22:43:04.167 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/2 --op log --pgid 1.0 2020-03-11T22:43:04.616 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:43:04.613+0000 7f910141fec0 -1 osd.0 39 log_to_monitors {default=true} 2020-03-11T22:43:04.978 INFO:tasks.workunit.client.0.smithi059.stderr:PG '1.0' not found 2020-03-11T22:43:05.499 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: return 1 2020-03-11T22:43:05.499 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:334: TEST_divergent_ec: activate_osd td/divergent-priors 2 2020-03-11T22:43:05.499 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:43:05.499 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:43:05.499 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=2 2020-03-11T22:43:05.500 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:43:05.500 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/2 2020-03-11T22:43:05.500 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=cb7f2e42-1934-4145-a3c3-29bdd93f8adb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:43:05.500 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:43:05.500 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:43:05.500 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:43:05.500 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-11T22:43:05.501 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-11T22:43:05.501 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:43:05.501 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:43:05.501 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:43:05.501 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:43:05.501 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:43:05.501 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:43:05.502 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:43:05.502 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:43:05.502 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:43:05.502 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17931/$cluster-$name.asok' 2020-03-11T22:43:05.502 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' 2020-03-11T22:43:05.502 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:43:05.503 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:43:05.503 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:43:05.503 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:43:05.503 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:43:05.503 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:43:05.503 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:43:05.503 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:43:05.504 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/2 2020-03-11T22:43:05.505 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.2 2020-03-11T22:43:05.505 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 2 --fsid=cb7f2e42-1934-4145-a3c3-29bdd93f8adb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:43:05.505 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/2/whoami 2020-03-11T22:43:05.505 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 2 = 2 ']' 2020-03-11T22:43:05.505 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:43:05.505 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:43:05.506 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:43:05.506 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.2 2020-03-11T22:43:05.531 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:43:05.528+0000 7f6c14453ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:05.533 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:43:05.530+0000 7f6c14453ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:05.534 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:43:05.531+0000 7f6c14453ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:05.876 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 2 2020-03-11T22:43:05.876 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:43:05.876 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:43:05.876 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:43:05.876 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:43:05.877 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:43:05.877 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:43:05.877 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:43:05.877 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:43:05.877 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:43:06.056 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:43:06.053+0000 7f6c14453ec0 -1 Falling back to public interface 2020-03-11T22:43:06.265 INFO:tasks.workunit.client.0.smithi059.stdout:osd.2 up in weight 1 up_from 36 up_thru 0 down_at 32 last_clean_interval [14,28) [v2:127.0.0.1:6810/49338,v1:127.0.0.1:6811/49338] [v2:127.0.0.1:6812/49338,v1:127.0.0.1:6813/49338] exists,up d2482f9a-6ef9-4322-9e82-b5d8b8810499 2020-03-11T22:43:06.265 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:43:06.266 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:43:06.266 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:43:06.266 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:336: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 1 --op log --pgid 1.0 2020-03-11T22:43:06.266 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/divergent-priors 2020-03-11T22:43:06.266 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-03-11T22:43:06.267 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=1 2020-03-11T22:43:06.267 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-03-11T22:43:06.267 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2020-03-11T22:43:06.267 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op log --pgid 1.0 2020-03-11T22:43:07.018 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:43:07.015+0000 7f6c14453ec0 -1 osd.2 39 log_to_monitors {default=true} 2020-03-11T22:43:07.046 INFO:tasks.workunit.client.0.smithi059.stderr:PG '1.0' not found 2020-03-11T22:43:07.566 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: return 1 2020-03-11T22:43:07.567 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:338: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:43:07.567 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:338: TEST_divergent_ec: wait_for_clean 2020-03-11T22:43:07.567 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:43:07.567 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:43:07.567 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:43:07.567 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:43:07.568 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:43:07.568 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:43:07.569 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:43:07.569 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:43:07.569 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:43:07.569 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:43:07.637 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:43:07.637 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:43:07.637 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:43:07.638 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:43:07.638 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:43:08.017 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:43:08.017 INFO:tasks.workunit.client.0.smithi059.stderr:1 2020-03-11T22:43:08.017 INFO:tasks.workunit.client.0.smithi059.stderr:2' 2020-03-11T22:43:08.018 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:43:08.018 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:43:08.018 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:43:08.153 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=176093659139 2020-03-11T22:43:08.153 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-176093659139' 2020-03-11T22:43:08.154 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:43:08.154 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:43:08.248 INFO:tasks.workunit.client.0.smithi059.stderr:Error ENXIO: problem getting command descriptions from osd.1 2020-03-11T22:43:08.256 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq= 2020-03-11T22:43:08.257 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-176093659139 1-' 2020-03-11T22:43:08.257 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:43:08.257 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:43:08.392 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=188978561027 2020-03-11T22:43:08.393 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-176093659139 1- 2-188978561027' 2020-03-11T22:43:08.393 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:43:08.393 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-176093659139 2020-03-11T22:43:08.394 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:43:08.394 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:43:08.394 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-176093659139 2020-03-11T22:43:08.395 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:43:08.395 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=176093659139 2020-03-11T22:43:08.396 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 176093659139' 2020-03-11T22:43:08.396 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:43:08.396 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.0 seq 176093659139 2020-03-11T22:43:08.763 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 1 -lt 176093659139 2020-03-11T22:43:08.763 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:43:09.764 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:43:09.765 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:43:10.142 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 176093659139 -lt 176093659139 2020-03-11T22:43:10.143 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:43:10.143 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1- 2020-03-11T22:43:10.143 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:43:10.144 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:43:10.144 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1- 2020-03-11T22:43:10.145 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:43:10.146 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq= 2020-03-11T22:43:10.146 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq ' 2020-03-11T22:43:10.146 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:43:10.146 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.1 seq 2020-03-11T22:43:10.528 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672978 -lt 2020-03-11T22:43:10.528 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 2079: test: 42949672978: unary operator expected 2020-03-11T22:43:10.529 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:43:10.529 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-188978561027 2020-03-11T22:43:10.529 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:43:10.530 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:43:10.530 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-188978561027 2020-03-11T22:43:10.530 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:43:10.531 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=188978561027 2020-03-11T22:43:10.531 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 188978561027' 2020-03-11T22:43:10.532 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:43:10.532 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.2 seq 188978561027 2020-03-11T22:43:10.892 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 188978561027 -lt 188978561027 2020-03-11T22:43:10.893 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:43:10.893 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:43:10.894 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:43:11.367 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:43:11.367 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:43:11.367 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:43:11.367 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:43:11.367 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:43:11.367 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:43:11.368 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:43:11.368 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:43:11.722 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:43:11.722 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:43:11.723 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:43:11.723 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:43:12.209 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:43:12.210 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:43:12.210 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:43:12.210 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:43:12.210 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:43:12.210 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:43:12.311 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:43:12.311 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:43:12.311 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:43:12.311 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:43:12.311 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:43:12.311 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:43:12.312 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:43:12.312 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:43:12.658 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:43:12.659 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:43:12.659 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:43:12.659 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:43:13.129 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:43:13.129 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:43:13.129 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:43:13.129 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:43:13.129 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:43:13.130 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:43:13.130 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:43:13.130 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:43:13.130 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:43:13.722 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:43:13.722 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:43:13.723 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 1 >= 8 )) 2020-03-11T22:43:13.723 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:43:13.723 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.2 2020-03-11T22:43:13.923 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:43:13.923 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:43:13.924 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:43:13.924 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:43:13.924 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:43:13.925 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:43:13.925 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:43:13.925 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:43:14.279 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:43:14.280 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:43:14.280 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:43:14.280 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:43:14.766 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:43:14.766 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:43:14.766 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:43:14.767 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:43:14.767 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:43:14.767 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:43:14.767 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:43:14.767 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:43:14.767 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:43:15.318 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:43:15.318 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:43:15.318 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 2 >= 8 )) 2020-03-11T22:43:15.318 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:43:15.319 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.4 2020-03-11T22:43:15.718 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:43:15.719 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:43:15.719 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:43:15.720 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:43:15.720 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:43:15.720 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:43:15.720 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:43:15.720 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:43:16.069 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:43:16.070 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:43:16.070 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:43:16.070 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:43:16.542 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:43:16.542 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:43:16.542 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:43:16.542 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:43:16.543 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:43:16.543 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:43:16.543 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:43:16.543 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:43:16.543 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:43:17.013 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:43:17.014 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:43:17.014 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 3 >= 8 )) 2020-03-11T22:43:17.014 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:43:17.014 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.8 2020-03-11T22:43:17.814 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:43:17.815 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:43:17.816 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:43:17.816 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:43:17.816 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:43:17.816 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:43:17.816 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:43:17.816 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:43:18.185 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:43:18.185 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:43:18.186 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:43:18.186 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:43:18.655 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:43:18.655 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:43:18.656 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:43:18.656 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:43:18.656 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:43:18.656 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:43:18.656 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:43:18.656 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:43:18.657 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:43:19.127 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:43:19.127 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:43:19.127 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 4 >= 8 )) 2020-03-11T22:43:19.127 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:43:19.127 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 1.6 2020-03-11T22:43:20.728 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:43:20.728 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:43:20.729 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:43:20.729 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:43:20.729 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:43:20.729 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:43:20.730 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:43:20.730 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:43:21.089 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:43:21.090 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:43:21.090 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:43:21.090 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:43:21.586 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:43:21.586 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:43:21.586 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:43:21.586 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:43:21.586 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:43:21.586 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:43:21.587 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:43:21.587 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:43:21.587 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:43:22.068 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:43:22.069 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:43:22.069 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 5 >= 8 )) 2020-03-11T22:43:22.069 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:43:22.069 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 3.2 2020-03-11T22:43:25.269 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:43:25.269 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:43:25.270 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:43:25.271 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:43:25.271 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:43:25.271 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:43:25.271 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:43:25.271 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:43:25.629 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:43:25.630 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:43:25.630 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:43:25.630 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:43:26.100 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:43:26.100 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:43:26.101 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:43:26.101 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:43:26.101 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:43:26.101 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:43:26.101 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:43:26.102 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:43:26.102 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:43:26.576 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:43:26.577 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:43:26.577 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 6 >= 8 )) 2020-03-11T22:43:26.577 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:43:26.577 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 6.4 2020-03-11T22:43:32.977 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:43:32.977 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:43:32.978 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:43:32.978 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:43:32.978 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:43:32.979 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:43:32.979 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:43:32.979 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:43:33.342 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:43:33.343 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:43:33.343 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:43:33.344 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:43:33.813 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:43:33.814 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:43:33.814 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:43:33.814 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:43:33.814 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:43:33.814 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:43:33.814 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:43:33.815 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:43:33.815 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:43:34.285 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:43:34.286 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:43:34.286 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 7 >= 8 )) 2020-03-11T22:43:34.286 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:43:34.286 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 7.3 2020-03-11T22:43:41.587 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:43:41.587 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:43:41.587 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:43:41.588 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:43:41.588 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:43:41.588 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:43:41.588 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:43:41.588 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:43:41.949 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:43:41.949 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:43:41.950 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:43:41.950 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:43:42.423 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:43:42.423 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:43:42.423 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:43:42.423 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:43:42.423 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:43:42.424 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:43:42.424 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:43:42.424 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:43:42.424 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:43:42.894 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:43:42.895 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:43:42.895 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 8 >= 8 )) 2020-03-11T22:43:42.895 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: ceph report 2020-03-11T22:43:43.403 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2020-03-11T22:43:43.403 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_fingerprint": "c0ec5b4a-1e22-41f2-aed6-87f25ac30cdf", 2020-03-11T22:43:43.403 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "15.1.0-1941-g4acc112", 2020-03-11T22:43:43.403 INFO:tasks.workunit.client.0.smithi059.stdout: "commit": "4acc1121e47039a3cdf2978ae458c10d153aa141", 2020-03-11T22:43:43.404 INFO:tasks.workunit.client.0.smithi059.stdout: "timestamp": "2020-03-11T22:43:43.399963+0000", 2020-03-11T22:43:43.404 INFO:tasks.workunit.client.0.smithi059.stdout: "tag": "", 2020-03-11T22:43:43.404 INFO:tasks.workunit.client.0.smithi059.stdout: "health": { 2020-03-11T22:43:43.404 INFO:tasks.workunit.client.0.smithi059.stdout: "status": "HEALTH_WARN", 2020-03-11T22:43:43.404 INFO:tasks.workunit.client.0.smithi059.stdout: "checks": { 2020-03-11T22:43:43.404 INFO:tasks.workunit.client.0.smithi059.stdout: "OSDMAP_FLAGS": { 2020-03-11T22:43:43.404 INFO:tasks.workunit.client.0.smithi059.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:43:43.405 INFO:tasks.workunit.client.0.smithi059.stdout: "summary": { 2020-03-11T22:43:43.405 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "nodown,noout,noin flag(s) set", 2020-03-11T22:43:43.405 INFO:tasks.workunit.client.0.smithi059.stdout: "count": 17 2020-03-11T22:43:43.405 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.405 INFO:tasks.workunit.client.0.smithi059.stdout: "detail": [], 2020-03-11T22:43:43.405 INFO:tasks.workunit.client.0.smithi059.stdout: "muted": false 2020-03-11T22:43:43.405 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.405 INFO:tasks.workunit.client.0.smithi059.stdout: "OSD_DOWN": { 2020-03-11T22:43:43.406 INFO:tasks.workunit.client.0.smithi059.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:43:43.406 INFO:tasks.workunit.client.0.smithi059.stdout: "summary": { 2020-03-11T22:43:43.406 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "1 osds down", 2020-03-11T22:43:43.406 INFO:tasks.workunit.client.0.smithi059.stdout: "count": 1 2020-03-11T22:43:43.406 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.406 INFO:tasks.workunit.client.0.smithi059.stdout: "detail": [ 2020-03-11T22:43:43.406 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.407 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "osd.1 (root=default,host=smithi059) is down" 2020-03-11T22:43:43.407 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:43.407 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:43:43.407 INFO:tasks.workunit.client.0.smithi059.stdout: "muted": false 2020-03-11T22:43:43.407 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.407 INFO:tasks.workunit.client.0.smithi059.stdout: "PG_DEGRADED": { 2020-03-11T22:43:43.407 INFO:tasks.workunit.client.0.smithi059.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:43:43.408 INFO:tasks.workunit.client.0.smithi059.stdout: "summary": { 2020-03-11T22:43:43.408 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "Degraded data redundancy: 101/303 objects degraded (33.333%), 1 pg degraded", 2020-03-11T22:43:43.408 INFO:tasks.workunit.client.0.smithi059.stdout: "count": 1 2020-03-11T22:43:43.408 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.408 INFO:tasks.workunit.client.0.smithi059.stdout: "detail": [ 2020-03-11T22:43:43.408 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.408 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [2147483647,0,2]" 2020-03-11T22:43:43.409 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:43.409 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:43:43.409 INFO:tasks.workunit.client.0.smithi059.stdout: "muted": false 2020-03-11T22:43:43.409 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.409 INFO:tasks.workunit.client.0.smithi059.stdout: "POOL_APP_NOT_ENABLED": { 2020-03-11T22:43:43.409 INFO:tasks.workunit.client.0.smithi059.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:43:43.409 INFO:tasks.workunit.client.0.smithi059.stdout: "summary": { 2020-03-11T22:43:43.410 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "1 pool(s) do not have an application enabled", 2020-03-11T22:43:43.410 INFO:tasks.workunit.client.0.smithi059.stdout: "count": 1 2020-03-11T22:43:43.410 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.410 INFO:tasks.workunit.client.0.smithi059.stdout: "detail": [ 2020-03-11T22:43:43.410 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.410 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "application not enabled on pool 'test'" 2020-03-11T22:43:43.410 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.411 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.411 INFO:tasks.workunit.client.0.smithi059.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2020-03-11T22:43:43.411 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:43.411 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:43:43.411 INFO:tasks.workunit.client.0.smithi059.stdout: "muted": false 2020-03-11T22:43:43.411 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:43.411 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.412 INFO:tasks.workunit.client.0.smithi059.stdout: "mutes": [] 2020-03-11T22:43:43.412 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.412 INFO:tasks.workunit.client.0.smithi059.stdout: "monmap_first_committed": 1, 2020-03-11T22:43:43.412 INFO:tasks.workunit.client.0.smithi059.stdout: "monmap_last_committed": 1, 2020-03-11T22:43:43.412 INFO:tasks.workunit.client.0.smithi059.stdout: "monmap": { 2020-03-11T22:43:43.412 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": 1, 2020-03-11T22:43:43.412 INFO:tasks.workunit.client.0.smithi059.stdout: "fsid": "cb7f2e42-1934-4145-a3c3-29bdd93f8adb", 2020-03-11T22:43:43.412 INFO:tasks.workunit.client.0.smithi059.stdout: "modified": "2020-03-11T22:41:10.071517Z", 2020-03-11T22:43:43.413 INFO:tasks.workunit.client.0.smithi059.stdout: "created": "2020-03-11T22:41:10.071517Z", 2020-03-11T22:43:43.413 INFO:tasks.workunit.client.0.smithi059.stdout: "min_mon_release": 15, 2020-03-11T22:43:43.413 INFO:tasks.workunit.client.0.smithi059.stdout: "min_mon_release_name": "octopus", 2020-03-11T22:43:43.413 INFO:tasks.workunit.client.0.smithi059.stdout: "features": { 2020-03-11T22:43:43.413 INFO:tasks.workunit.client.0.smithi059.stdout: "persistent": [ 2020-03-11T22:43:43.413 INFO:tasks.workunit.client.0.smithi059.stdout: "kraken", 2020-03-11T22:43:43.413 INFO:tasks.workunit.client.0.smithi059.stdout: "luminous", 2020-03-11T22:43:43.414 INFO:tasks.workunit.client.0.smithi059.stdout: "mimic", 2020-03-11T22:43:43.414 INFO:tasks.workunit.client.0.smithi059.stdout: "osdmap-prune", 2020-03-11T22:43:43.414 INFO:tasks.workunit.client.0.smithi059.stdout: "nautilus", 2020-03-11T22:43:43.414 INFO:tasks.workunit.client.0.smithi059.stdout: "octopus" 2020-03-11T22:43:43.414 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:43:43.414 INFO:tasks.workunit.client.0.smithi059.stdout: "optional": [] 2020-03-11T22:43:43.414 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.415 INFO:tasks.workunit.client.0.smithi059.stdout: "mons": [ 2020-03-11T22:43:43.415 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.415 INFO:tasks.workunit.client.0.smithi059.stdout: "rank": 0, 2020-03-11T22:43:43.415 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "a", 2020-03-11T22:43:43.415 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addrs": { 2020-03-11T22:43:43.415 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:43:43.415 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.415 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:43:43.416 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:7115", 2020-03-11T22:43:43.416 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 0 2020-03-11T22:43:43.416 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:43.416 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:43:43.416 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.416 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:7115/0", 2020-03-11T22:43:43.416 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addr": "127.0.0.1:7115/0", 2020-03-11T22:43:43.417 INFO:tasks.workunit.client.0.smithi059.stdout: "priority": 0, 2020-03-11T22:43:43.417 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 0 2020-03-11T22:43:43.417 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:43.417 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:43:43.417 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.417 INFO:tasks.workunit.client.0.smithi059.stdout: "quorum": [ 2020-03-11T22:43:43.417 INFO:tasks.workunit.client.0.smithi059.stdout: 0 2020-03-11T22:43:43.418 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:43:43.418 INFO:tasks.workunit.client.0.smithi059.stdout: "osdmap": { 2020-03-11T22:43:43.418 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": 45, 2020-03-11T22:43:43.418 INFO:tasks.workunit.client.0.smithi059.stdout: "fsid": "cb7f2e42-1934-4145-a3c3-29bdd93f8adb", 2020-03-11T22:43:43.418 INFO:tasks.workunit.client.0.smithi059.stdout: "created": "2020-03-11T22:41:10.840470+0000", 2020-03-11T22:43:43.418 INFO:tasks.workunit.client.0.smithi059.stdout: "modified": "2020-03-11T22:43:07.578434+0000", 2020-03-11T22:43:43.418 INFO:tasks.workunit.client.0.smithi059.stdout: "last_up_change": "2020-03-11T22:43:07.311291+0000", 2020-03-11T22:43:43.419 INFO:tasks.workunit.client.0.smithi059.stdout: "last_in_change": "2020-03-11T22:42:20.779637+0000", 2020-03-11T22:43:43.419 INFO:tasks.workunit.client.0.smithi059.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-03-11T22:43:43.419 INFO:tasks.workunit.client.0.smithi059.stdout: "flags_num": 5800384, 2020-03-11T22:43:43.419 INFO:tasks.workunit.client.0.smithi059.stdout: "flags_set": [ 2020-03-11T22:43:43.419 INFO:tasks.workunit.client.0.smithi059.stdout: "nodown", 2020-03-11T22:43:43.419 INFO:tasks.workunit.client.0.smithi059.stdout: "noin", 2020-03-11T22:43:43.419 INFO:tasks.workunit.client.0.smithi059.stdout: "noout", 2020-03-11T22:43:43.419 INFO:tasks.workunit.client.0.smithi059.stdout: "pglog_hardlimit", 2020-03-11T22:43:43.420 INFO:tasks.workunit.client.0.smithi059.stdout: "purged_snapdirs", 2020-03-11T22:43:43.420 INFO:tasks.workunit.client.0.smithi059.stdout: "recovery_deletes", 2020-03-11T22:43:43.420 INFO:tasks.workunit.client.0.smithi059.stdout: "sortbitwise" 2020-03-11T22:43:43.420 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:43:43.420 INFO:tasks.workunit.client.0.smithi059.stdout: "crush_version": 8, 2020-03-11T22:43:43.420 INFO:tasks.workunit.client.0.smithi059.stdout: "full_ratio": 0.99000000953674316, 2020-03-11T22:43:43.420 INFO:tasks.workunit.client.0.smithi059.stdout: "backfillfull_ratio": 0.99000000953674316, 2020-03-11T22:43:43.421 INFO:tasks.workunit.client.0.smithi059.stdout: "nearfull_ratio": 0.85000002384185791, 2020-03-11T22:43:43.421 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_snapshot": "", 2020-03-11T22:43:43.421 INFO:tasks.workunit.client.0.smithi059.stdout: "pool_max": 1, 2020-03-11T22:43:43.421 INFO:tasks.workunit.client.0.smithi059.stdout: "max_osd": 3, 2020-03-11T22:43:43.421 INFO:tasks.workunit.client.0.smithi059.stdout: "require_min_compat_client": "jewel", 2020-03-11T22:43:43.421 INFO:tasks.workunit.client.0.smithi059.stdout: "min_compat_client": "jewel", 2020-03-11T22:43:43.421 INFO:tasks.workunit.client.0.smithi059.stdout: "require_osd_release": "octopus", 2020-03-11T22:43:43.422 INFO:tasks.workunit.client.0.smithi059.stdout: "pools": [ 2020-03-11T22:43:43.422 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.422 INFO:tasks.workunit.client.0.smithi059.stdout: "pool": 1, 2020-03-11T22:43:43.422 INFO:tasks.workunit.client.0.smithi059.stdout: "pool_name": "test", 2020-03-11T22:43:43.422 INFO:tasks.workunit.client.0.smithi059.stdout: "create_time": "2020-03-11T22:41:33.873560+0000", 2020-03-11T22:43:43.422 INFO:tasks.workunit.client.0.smithi059.stdout: "flags": 16389, 2020-03-11T22:43:43.422 INFO:tasks.workunit.client.0.smithi059.stdout: "flags_names": "hashpspool,ec_overwrites,pool_snaps", 2020-03-11T22:43:43.423 INFO:tasks.workunit.client.0.smithi059.stdout: "type": 3, 2020-03-11T22:43:43.423 INFO:tasks.workunit.client.0.smithi059.stdout: "size": 3, 2020-03-11T22:43:43.423 INFO:tasks.workunit.client.0.smithi059.stdout: "min_size": 2, 2020-03-11T22:43:43.423 INFO:tasks.workunit.client.0.smithi059.stdout: "crush_rule": 1, 2020-03-11T22:43:43.423 INFO:tasks.workunit.client.0.smithi059.stdout: "object_hash": 2, 2020-03-11T22:43:43.423 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_autoscale_mode": "off", 2020-03-11T22:43:43.423 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_num": 1, 2020-03-11T22:43:43.423 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_placement_num": 1, 2020-03-11T22:43:43.424 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_placement_num_target": 1, 2020-03-11T22:43:43.424 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_num_target": 1, 2020-03-11T22:43:43.424 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_num_pending": 1, 2020-03-11T22:43:43.424 INFO:tasks.workunit.client.0.smithi059.stdout: "last_pg_merge_meta": { 2020-03-11T22:43:43.424 INFO:tasks.workunit.client.0.smithi059.stdout: "source_pgid": "0.0", 2020-03-11T22:43:43.424 INFO:tasks.workunit.client.0.smithi059.stdout: "ready_epoch": 0, 2020-03-11T22:43:43.424 INFO:tasks.workunit.client.0.smithi059.stdout: "last_epoch_started": 0, 2020-03-11T22:43:43.425 INFO:tasks.workunit.client.0.smithi059.stdout: "last_epoch_clean": 0, 2020-03-11T22:43:43.425 INFO:tasks.workunit.client.0.smithi059.stdout: "source_version": "0'0", 2020-03-11T22:43:43.425 INFO:tasks.workunit.client.0.smithi059.stdout: "target_version": "0'0" 2020-03-11T22:43:43.425 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.425 INFO:tasks.workunit.client.0.smithi059.stdout: "last_change": "29", 2020-03-11T22:43:43.425 INFO:tasks.workunit.client.0.smithi059.stdout: "last_force_op_resend": "0", 2020-03-11T22:43:43.425 INFO:tasks.workunit.client.0.smithi059.stdout: "last_force_op_resend_prenautilus": "0", 2020-03-11T22:43:43.426 INFO:tasks.workunit.client.0.smithi059.stdout: "last_force_op_resend_preluminous": "0", 2020-03-11T22:43:43.426 INFO:tasks.workunit.client.0.smithi059.stdout: "auid": 0, 2020-03-11T22:43:43.426 INFO:tasks.workunit.client.0.smithi059.stdout: "snap_mode": "pool", 2020-03-11T22:43:43.426 INFO:tasks.workunit.client.0.smithi059.stdout: "snap_seq": 1, 2020-03-11T22:43:43.426 INFO:tasks.workunit.client.0.smithi059.stdout: "snap_epoch": 29, 2020-03-11T22:43:43.426 INFO:tasks.workunit.client.0.smithi059.stdout: "pool_snaps": [ 2020-03-11T22:43:43.426 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.427 INFO:tasks.workunit.client.0.smithi059.stdout: "snapid": 1, 2020-03-11T22:43:43.427 INFO:tasks.workunit.client.0.smithi059.stdout: "stamp": "2020-03-11T22:42:01.471185+0000", 2020-03-11T22:43:43.427 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "snap1" 2020-03-11T22:43:43.427 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:43.427 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:43:43.427 INFO:tasks.workunit.client.0.smithi059.stdout: "removed_snaps": "[]", 2020-03-11T22:43:43.427 INFO:tasks.workunit.client.0.smithi059.stdout: "quota_max_bytes": 0, 2020-03-11T22:43:43.428 INFO:tasks.workunit.client.0.smithi059.stdout: "quota_max_objects": 0, 2020-03-11T22:43:43.428 INFO:tasks.workunit.client.0.smithi059.stdout: "tiers": [], 2020-03-11T22:43:43.428 INFO:tasks.workunit.client.0.smithi059.stdout: "tier_of": -1, 2020-03-11T22:43:43.428 INFO:tasks.workunit.client.0.smithi059.stdout: "read_tier": -1, 2020-03-11T22:43:43.428 INFO:tasks.workunit.client.0.smithi059.stdout: "write_tier": -1, 2020-03-11T22:43:43.428 INFO:tasks.workunit.client.0.smithi059.stdout: "cache_mode": "none", 2020-03-11T22:43:43.428 INFO:tasks.workunit.client.0.smithi059.stdout: "target_max_bytes": 0, 2020-03-11T22:43:43.429 INFO:tasks.workunit.client.0.smithi059.stdout: "target_max_objects": 0, 2020-03-11T22:43:43.429 INFO:tasks.workunit.client.0.smithi059.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-03-11T22:43:43.429 INFO:tasks.workunit.client.0.smithi059.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-03-11T22:43:43.429 INFO:tasks.workunit.client.0.smithi059.stdout: "cache_target_full_ratio_micro": 800000, 2020-03-11T22:43:43.429 INFO:tasks.workunit.client.0.smithi059.stdout: "cache_min_flush_age": 0, 2020-03-11T22:43:43.429 INFO:tasks.workunit.client.0.smithi059.stdout: "cache_min_evict_age": 0, 2020-03-11T22:43:43.429 INFO:tasks.workunit.client.0.smithi059.stdout: "erasure_code_profile": "myprofile", 2020-03-11T22:43:43.429 INFO:tasks.workunit.client.0.smithi059.stdout: "hit_set_params": { 2020-03-11T22:43:43.430 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "none" 2020-03-11T22:43:43.430 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.430 INFO:tasks.workunit.client.0.smithi059.stdout: "hit_set_period": 0, 2020-03-11T22:43:43.430 INFO:tasks.workunit.client.0.smithi059.stdout: "hit_set_count": 0, 2020-03-11T22:43:43.430 INFO:tasks.workunit.client.0.smithi059.stdout: "use_gmt_hitset": true, 2020-03-11T22:43:43.430 INFO:tasks.workunit.client.0.smithi059.stdout: "min_read_recency_for_promote": 0, 2020-03-11T22:43:43.430 INFO:tasks.workunit.client.0.smithi059.stdout: "min_write_recency_for_promote": 0, 2020-03-11T22:43:43.431 INFO:tasks.workunit.client.0.smithi059.stdout: "hit_set_grade_decay_rate": 0, 2020-03-11T22:43:43.431 INFO:tasks.workunit.client.0.smithi059.stdout: "hit_set_search_last_n": 0, 2020-03-11T22:43:43.431 INFO:tasks.workunit.client.0.smithi059.stdout: "grade_table": [], 2020-03-11T22:43:43.431 INFO:tasks.workunit.client.0.smithi059.stdout: "stripe_width": 8192, 2020-03-11T22:43:43.431 INFO:tasks.workunit.client.0.smithi059.stdout: "expected_num_objects": 0, 2020-03-11T22:43:43.431 INFO:tasks.workunit.client.0.smithi059.stdout: "fast_read": false, 2020-03-11T22:43:43.431 INFO:tasks.workunit.client.0.smithi059.stdout: "options": {}, 2020-03-11T22:43:43.432 INFO:tasks.workunit.client.0.smithi059.stdout: "application_metadata": {} 2020-03-11T22:43:43.432 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:43.432 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:43:43.432 INFO:tasks.workunit.client.0.smithi059.stdout: "osds": [ 2020-03-11T22:43:43.432 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.432 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 0, 2020-03-11T22:43:43.432 INFO:tasks.workunit.client.0.smithi059.stdout: "uuid": "af7f49a6-1c6c-446f-802f-d448ee247966", 2020-03-11T22:43:43.433 INFO:tasks.workunit.client.0.smithi059.stdout: "up": 1, 2020-03-11T22:43:43.433 INFO:tasks.workunit.client.0.smithi059.stdout: "in": 1, 2020-03-11T22:43:43.433 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 1, 2020-03-11T22:43:43.433 INFO:tasks.workunit.client.0.smithi059.stdout: "primary_affinity": 1, 2020-03-11T22:43:43.433 INFO:tasks.workunit.client.0.smithi059.stdout: "last_clean_begin": 35, 2020-03-11T22:43:43.433 INFO:tasks.workunit.client.0.smithi059.stdout: "last_clean_end": 39, 2020-03-11T22:43:43.433 INFO:tasks.workunit.client.0.smithi059.stdout: "up_from": 41, 2020-03-11T22:43:43.434 INFO:tasks.workunit.client.0.smithi059.stdout: "up_thru": 44, 2020-03-11T22:43:43.434 INFO:tasks.workunit.client.0.smithi059.stdout: "down_at": 40, 2020-03-11T22:43:43.434 INFO:tasks.workunit.client.0.smithi059.stdout: "lost_at": 0, 2020-03-11T22:43:43.434 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addrs": { 2020-03-11T22:43:43.434 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:43:43.434 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.434 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:43:43.434 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6802", 2020-03-11T22:43:43.435 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 50787 2020-03-11T22:43:43.435 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.435 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.435 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:43:43.435 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6803", 2020-03-11T22:43:43.435 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 50787 2020-03-11T22:43:43.435 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:43.436 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:43:43.436 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.436 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_addrs": { 2020-03-11T22:43:43.436 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:43:43.436 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.436 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:43:43.436 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6804", 2020-03-11T22:43:43.437 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 50787 2020-03-11T22:43:43.437 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.437 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.437 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:43:43.437 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6805", 2020-03-11T22:43:43.437 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 50787 2020-03-11T22:43:43.437 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:43.438 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:43:43.438 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.438 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_back_addrs": { 2020-03-11T22:43:43.438 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:43:43.438 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.438 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:43:43.438 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6808", 2020-03-11T22:43:43.439 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 50787 2020-03-11T22:43:43.439 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.439 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.439 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:43:43.439 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6809", 2020-03-11T22:43:43.439 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 50787 2020-03-11T22:43:43.439 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:43.440 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:43:43.440 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.440 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_front_addrs": { 2020-03-11T22:43:43.440 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:43:43.440 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.440 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:43:43.440 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6806", 2020-03-11T22:43:43.441 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 50787 2020-03-11T22:43:43.441 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.441 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.441 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:43:43.441 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6807", 2020-03-11T22:43:43.441 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 50787 2020-03-11T22:43:43.441 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:43.442 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:43:43.442 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.442 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addr": "127.0.0.1:6803/50787", 2020-03-11T22:43:43.442 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_addr": "127.0.0.1:6805/50787", 2020-03-11T22:43:43.442 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_back_addr": "127.0.0.1:6809/50787", 2020-03-11T22:43:43.442 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_front_addr": "127.0.0.1:6807/50787", 2020-03-11T22:43:43.443 INFO:tasks.workunit.client.0.smithi059.stdout: "state": [ 2020-03-11T22:43:43.443 INFO:tasks.workunit.client.0.smithi059.stdout: "exists", 2020-03-11T22:43:43.443 INFO:tasks.workunit.client.0.smithi059.stdout: "up" 2020-03-11T22:43:43.443 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:43:43.443 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.443 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.443 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 1, 2020-03-11T22:43:43.444 INFO:tasks.workunit.client.0.smithi059.stdout: "uuid": "01a46a03-2514-4118-9d87-d9feba7ccc07", 2020-03-11T22:43:43.444 INFO:tasks.workunit.client.0.smithi059.stdout: "up": 0, 2020-03-11T22:43:43.444 INFO:tasks.workunit.client.0.smithi059.stdout: "in": 1, 2020-03-11T22:43:43.444 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 1, 2020-03-11T22:43:43.444 INFO:tasks.workunit.client.0.smithi059.stdout: "primary_affinity": 1, 2020-03-11T22:43:43.444 INFO:tasks.workunit.client.0.smithi059.stdout: "last_clean_begin": 0, 2020-03-11T22:43:43.444 INFO:tasks.workunit.client.0.smithi059.stdout: "last_clean_end": 0, 2020-03-11T22:43:43.444 INFO:tasks.workunit.client.0.smithi059.stdout: "up_from": 10, 2020-03-11T22:43:43.445 INFO:tasks.workunit.client.0.smithi059.stdout: "up_thru": 24, 2020-03-11T22:43:43.445 INFO:tasks.workunit.client.0.smithi059.stdout: "down_at": 31, 2020-03-11T22:43:43.445 INFO:tasks.workunit.client.0.smithi059.stdout: "lost_at": 0, 2020-03-11T22:43:43.445 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addrs": { 2020-03-11T22:43:43.445 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:43:43.445 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.446 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:43:43.446 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:43:43.446 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 44413 2020-03-11T22:43:43.446 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.446 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.446 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:43:43.446 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:43:43.446 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 44413 2020-03-11T22:43:43.447 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:43.447 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:43:43.447 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.447 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_addrs": { 2020-03-11T22:43:43.447 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:43:43.447 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.447 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:43:43.448 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:43:43.448 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 44413 2020-03-11T22:43:43.448 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.448 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.448 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:43:43.448 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:43:43.448 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 44413 2020-03-11T22:43:43.449 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:43.449 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:43:43.449 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.449 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_back_addrs": { 2020-03-11T22:43:43.449 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:43:43.449 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.450 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:43:43.450 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:43:43.450 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 44413 2020-03-11T22:43:43.450 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.450 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.450 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:43:43.450 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:43:43.451 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 44413 2020-03-11T22:43:43.451 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:43.451 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:43:43.451 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.451 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_front_addrs": { 2020-03-11T22:43:43.451 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:43:43.451 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.452 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:43:43.452 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:43:43.452 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 44413 2020-03-11T22:43:43.452 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.452 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.452 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:43:43.452 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:43:43.453 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 44413 2020-03-11T22:43:43.453 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:43.453 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:43:43.453 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.453 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addr": "127.0.0.1:6811/44413", 2020-03-11T22:43:43.453 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_addr": "127.0.0.1:6813/44413", 2020-03-11T22:43:43.454 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_back_addr": "127.0.0.1:6817/44413", 2020-03-11T22:43:43.454 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_front_addr": "127.0.0.1:6815/44413", 2020-03-11T22:43:43.454 INFO:tasks.workunit.client.0.smithi059.stdout: "state": [ 2020-03-11T22:43:43.454 INFO:tasks.workunit.client.0.smithi059.stdout: "exists" 2020-03-11T22:43:43.454 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:43:43.454 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.454 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.455 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 2, 2020-03-11T22:43:43.455 INFO:tasks.workunit.client.0.smithi059.stdout: "uuid": "d2482f9a-6ef9-4322-9e82-b5d8b8810499", 2020-03-11T22:43:43.455 INFO:tasks.workunit.client.0.smithi059.stdout: "up": 1, 2020-03-11T22:43:43.455 INFO:tasks.workunit.client.0.smithi059.stdout: "in": 1, 2020-03-11T22:43:43.455 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 1, 2020-03-11T22:43:43.455 INFO:tasks.workunit.client.0.smithi059.stdout: "primary_affinity": 1, 2020-03-11T22:43:43.455 INFO:tasks.workunit.client.0.smithi059.stdout: "last_clean_begin": 36, 2020-03-11T22:43:43.456 INFO:tasks.workunit.client.0.smithi059.stdout: "last_clean_end": 39, 2020-03-11T22:43:43.456 INFO:tasks.workunit.client.0.smithi059.stdout: "up_from": 44, 2020-03-11T22:43:43.456 INFO:tasks.workunit.client.0.smithi059.stdout: "up_thru": 0, 2020-03-11T22:43:43.456 INFO:tasks.workunit.client.0.smithi059.stdout: "down_at": 43, 2020-03-11T22:43:43.456 INFO:tasks.workunit.client.0.smithi059.stdout: "lost_at": 0, 2020-03-11T22:43:43.456 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addrs": { 2020-03-11T22:43:43.456 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:43:43.457 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.457 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:43:43.457 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:43:43.457 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 51019 2020-03-11T22:43:43.457 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.457 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.457 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:43:43.458 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:43:43.458 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 51019 2020-03-11T22:43:43.458 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:43.458 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:43:43.458 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.458 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_addrs": { 2020-03-11T22:43:43.458 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:43:43.459 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.459 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:43:43.459 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:43:43.459 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 51019 2020-03-11T22:43:43.459 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.459 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.460 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:43:43.460 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:43:43.460 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 51019 2020-03-11T22:43:43.460 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:43.460 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:43:43.460 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.460 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_back_addrs": { 2020-03-11T22:43:43.461 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:43:43.461 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.461 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:43:43.461 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:43:43.461 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 51019 2020-03-11T22:43:43.461 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.461 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.462 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:43:43.462 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:43:43.462 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 51019 2020-03-11T22:43:43.462 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:43.462 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:43:43.462 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.462 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_front_addrs": { 2020-03-11T22:43:43.462 INFO:tasks.workunit.client.0.smithi059.stdout: "addrvec": [ 2020-03-11T22:43:43.463 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.463 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v2", 2020-03-11T22:43:43.463 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:43:43.463 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 51019 2020-03-11T22:43:43.463 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.463 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.464 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "v1", 2020-03-11T22:43:43.464 INFO:tasks.workunit.client.0.smithi059.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:43:43.464 INFO:tasks.workunit.client.0.smithi059.stdout: "nonce": 51019 2020-03-11T22:43:43.464 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:43.464 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:43:43.464 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.464 INFO:tasks.workunit.client.0.smithi059.stdout: "public_addr": "127.0.0.1:6811/51019", 2020-03-11T22:43:43.465 INFO:tasks.workunit.client.0.smithi059.stdout: "cluster_addr": "127.0.0.1:6813/51019", 2020-03-11T22:43:43.465 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_back_addr": "127.0.0.1:6817/51019", 2020-03-11T22:43:43.465 INFO:tasks.workunit.client.0.smithi059.stdout: "heartbeat_front_addr": "127.0.0.1:6815/51019", 2020-03-11T22:43:43.465 INFO:tasks.workunit.client.0.smithi059.stdout: "state": [ 2020-03-11T22:43:43.465 INFO:tasks.workunit.client.0.smithi059.stdout: "exists", 2020-03-11T22:43:43.465 INFO:tasks.workunit.client.0.smithi059.stdout: "up" 2020-03-11T22:43:43.465 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:43:43.466 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:43.466 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:43:43.466 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_xinfo": [ 2020-03-11T22:43:43.466 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.466 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 0, 2020-03-11T22:43:43.466 INFO:tasks.workunit.client.0.smithi059.stdout: "down_stamp": "2020-03-11T22:43:04.705207+0000", 2020-03-11T22:43:43.466 INFO:tasks.workunit.client.0.smithi059.stdout: "laggy_probability": 0, 2020-03-11T22:43:43.467 INFO:tasks.workunit.client.0.smithi059.stdout: "laggy_interval": 0, 2020-03-11T22:43:43.467 INFO:tasks.workunit.client.0.smithi059.stdout: "features": 4540138292836696063, 2020-03-11T22:43:43.467 INFO:tasks.workunit.client.0.smithi059.stdout: "old_weight": 0, 2020-03-11T22:43:43.467 INFO:tasks.workunit.client.0.smithi059.stdout: "last_purged_snaps_scrub": "2020-03-11T22:41:17.873225+0000", 2020-03-11T22:43:43.467 INFO:tasks.workunit.client.0.smithi059.stdout: "dead_epoch": 0 2020-03-11T22:43:43.467 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.467 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.467 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 1, 2020-03-11T22:43:43.468 INFO:tasks.workunit.client.0.smithi059.stdout: "down_stamp": "2020-03-11T22:42:13.111456+0000", 2020-03-11T22:43:43.468 INFO:tasks.workunit.client.0.smithi059.stdout: "laggy_probability": 0, 2020-03-11T22:43:43.468 INFO:tasks.workunit.client.0.smithi059.stdout: "laggy_interval": 0, 2020-03-11T22:43:43.468 INFO:tasks.workunit.client.0.smithi059.stdout: "features": 4540138292836696063, 2020-03-11T22:43:43.468 INFO:tasks.workunit.client.0.smithi059.stdout: "old_weight": 0, 2020-03-11T22:43:43.468 INFO:tasks.workunit.client.0.smithi059.stdout: "last_purged_snaps_scrub": "2020-03-11T22:41:24.468813+0000", 2020-03-11T22:43:43.468 INFO:tasks.workunit.client.0.smithi059.stdout: "dead_epoch": 0 2020-03-11T22:43:43.469 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.469 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.469 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 2, 2020-03-11T22:43:43.469 INFO:tasks.workunit.client.0.smithi059.stdout: "down_stamp": "2020-03-11T22:43:07.102659+0000", 2020-03-11T22:43:43.469 INFO:tasks.workunit.client.0.smithi059.stdout: "laggy_probability": 0, 2020-03-11T22:43:43.469 INFO:tasks.workunit.client.0.smithi059.stdout: "laggy_interval": 0, 2020-03-11T22:43:43.470 INFO:tasks.workunit.client.0.smithi059.stdout: "features": 4540138292836696063, 2020-03-11T22:43:43.470 INFO:tasks.workunit.client.0.smithi059.stdout: "old_weight": 0, 2020-03-11T22:43:43.470 INFO:tasks.workunit.client.0.smithi059.stdout: "last_purged_snaps_scrub": "2020-03-11T22:41:30.922789+0000", 2020-03-11T22:43:43.470 INFO:tasks.workunit.client.0.smithi059.stdout: "dead_epoch": 0 2020-03-11T22:43:43.470 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:43.470 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:43:43.471 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_upmap": [], 2020-03-11T22:43:43.471 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_upmap_items": [], 2020-03-11T22:43:43.471 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_temp": [], 2020-03-11T22:43:43.471 INFO:tasks.workunit.client.0.smithi059.stdout: "primary_temp": [], 2020-03-11T22:43:43.471 INFO:tasks.workunit.client.0.smithi059.stdout: "blacklist": {}, 2020-03-11T22:43:43.471 INFO:tasks.workunit.client.0.smithi059.stdout: "erasure_code_profiles": { 2020-03-11T22:43:43.472 INFO:tasks.workunit.client.0.smithi059.stdout: "default": { 2020-03-11T22:43:43.472 INFO:tasks.workunit.client.0.smithi059.stdout: "k": "2", 2020-03-11T22:43:43.472 INFO:tasks.workunit.client.0.smithi059.stdout: "m": "2", 2020-03-11T22:43:43.472 INFO:tasks.workunit.client.0.smithi059.stdout: "plugin": "jerasure", 2020-03-11T22:43:43.472 INFO:tasks.workunit.client.0.smithi059.stdout: "technique": "reed_sol_van" 2020-03-11T22:43:43.472 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.472 INFO:tasks.workunit.client.0.smithi059.stdout: "myprofile": { 2020-03-11T22:43:43.472 INFO:tasks.workunit.client.0.smithi059.stdout: "crush-device-class": "", 2020-03-11T22:43:43.473 INFO:tasks.workunit.client.0.smithi059.stdout: "crush-failure-domain": "osd", 2020-03-11T22:43:43.473 INFO:tasks.workunit.client.0.smithi059.stdout: "crush-root": "default", 2020-03-11T22:43:43.473 INFO:tasks.workunit.client.0.smithi059.stdout: "jerasure-per-chunk-alignment": "false", 2020-03-11T22:43:43.473 INFO:tasks.workunit.client.0.smithi059.stdout: "k": "2", 2020-03-11T22:43:43.473 INFO:tasks.workunit.client.0.smithi059.stdout: "m": "1", 2020-03-11T22:43:43.473 INFO:tasks.workunit.client.0.smithi059.stdout: "plugin": "jerasure", 2020-03-11T22:43:43.474 INFO:tasks.workunit.client.0.smithi059.stdout: "technique": "reed_sol_van", 2020-03-11T22:43:43.474 INFO:tasks.workunit.client.0.smithi059.stdout: "w": "8" 2020-03-11T22:43:43.474 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:43.474 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.474 INFO:tasks.workunit.client.0.smithi059.stdout: "removed_snaps_queue": [], 2020-03-11T22:43:43.474 INFO:tasks.workunit.client.0.smithi059.stdout: "new_removed_snaps": [], 2020-03-11T22:43:43.474 INFO:tasks.workunit.client.0.smithi059.stdout: "new_purged_snaps": [], 2020-03-11T22:43:43.475 INFO:tasks.workunit.client.0.smithi059.stdout: "crush_node_flags": {}, 2020-03-11T22:43:43.475 INFO:tasks.workunit.client.0.smithi059.stdout: "device_class_flags": {} 2020-03-11T22:43:43.475 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.475 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_metadata": [ 2020-03-11T22:43:43.475 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.475 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 0, 2020-03-11T22:43:43.475 INFO:tasks.workunit.client.0.smithi059.stdout: "arch": "x86_64", 2020-03-11T22:43:43.476 INFO:tasks.workunit.client.0.smithi059.stdout: "back_addr": "[v2:127.0.0.1:6804/50787,v1:127.0.0.1:6805/50787]", 2020-03-11T22:43:43.476 INFO:tasks.workunit.client.0.smithi059.stdout: "back_iface": "", 2020-03-11T22:43:43.476 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs": "1", 2020-03-11T22:43:43.476 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:43:43.476 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:43:43.476 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:43:43.476 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:43:43.477 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:43:43.477 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:43:43.477 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:43:43.477 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2020-03-11T22:43:43.477 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:43:43.477 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:43:43.477 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:43:43.478 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:43:43.478 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_release": "octopus", 2020-03-11T22:43:43.478 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:43:43.478 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:43:43.478 INFO:tasks.workunit.client.0.smithi059.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-03-11T22:43:43.478 INFO:tasks.workunit.client.0.smithi059.stdout: "default_device_class": "hdd", 2020-03-11T22:43:43.478 INFO:tasks.workunit.client.0.smithi059.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4J2P7", 2020-03-11T22:43:43.479 INFO:tasks.workunit.client.0.smithi059.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:43:43.479 INFO:tasks.workunit.client.0.smithi059.stdout: "devices": "sda", 2020-03-11T22:43:43.479 INFO:tasks.workunit.client.0.smithi059.stdout: "distro": "centos", 2020-03-11T22:43:43.479 INFO:tasks.workunit.client.0.smithi059.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:43:43.479 INFO:tasks.workunit.client.0.smithi059.stdout: "distro_version": "8", 2020-03-11T22:43:43.479 INFO:tasks.workunit.client.0.smithi059.stdout: "front_addr": "[v2:127.0.0.1:6802/50787,v1:127.0.0.1:6803/50787]", 2020-03-11T22:43:43.480 INFO:tasks.workunit.client.0.smithi059.stdout: "front_iface": "", 2020-03-11T22:43:43.480 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/50787,v1:127.0.0.1:6809/50787]", 2020-03-11T22:43:43.480 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/50787,v1:127.0.0.1:6807/50787]", 2020-03-11T22:43:43.480 INFO:tasks.workunit.client.0.smithi059.stdout: "hostname": "smithi059", 2020-03-11T22:43:43.480 INFO:tasks.workunit.client.0.smithi059.stdout: "journal_rotational": "1", 2020-03-11T22:43:43.480 INFO:tasks.workunit.client.0.smithi059.stdout: "kernel_description": "#1 SMP Wed Feb 5 02:00:39 UTC 2020", 2020-03-11T22:43:43.480 INFO:tasks.workunit.client.0.smithi059.stdout: "kernel_version": "4.18.0-147.5.1.el8_1.x86_64", 2020-03-11T22:43:43.481 INFO:tasks.workunit.client.0.smithi059.stdout: "mem_swap_kb": "0", 2020-03-11T22:43:43.481 INFO:tasks.workunit.client.0.smithi059.stdout: "mem_total_kb": "32662084", 2020-03-11T22:43:43.481 INFO:tasks.workunit.client.0.smithi059.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:43:43.481 INFO:tasks.workunit.client.0.smithi059.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:43:43.481 INFO:tasks.workunit.client.0.smithi059.stdout: "os": "Linux", 2020-03-11T22:43:43.481 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_data": "td/divergent-priors/0", 2020-03-11T22:43:43.481 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:43:43.482 INFO:tasks.workunit.client.0.smithi059.stdout: "rotational": "1" 2020-03-11T22:43:43.482 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.482 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.482 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 1, 2020-03-11T22:43:43.482 INFO:tasks.workunit.client.0.smithi059.stdout: "arch": "x86_64", 2020-03-11T22:43:43.482 INFO:tasks.workunit.client.0.smithi059.stdout: "back_addr": "[v2:127.0.0.1:6812/44413,v1:127.0.0.1:6813/44413]", 2020-03-11T22:43:43.482 INFO:tasks.workunit.client.0.smithi059.stdout: "back_iface": "", 2020-03-11T22:43:43.483 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs": "1", 2020-03-11T22:43:43.483 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:43:43.483 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:43:43.483 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:43:43.483 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:43:43.483 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:43:43.483 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:43:43.484 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:43:43.484 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2020-03-11T22:43:43.484 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:43:43.484 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:43:43.484 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:43:43.484 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:43:43.485 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_release": "octopus", 2020-03-11T22:43:43.485 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:43:43.485 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:43:43.485 INFO:tasks.workunit.client.0.smithi059.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-03-11T22:43:43.485 INFO:tasks.workunit.client.0.smithi059.stdout: "default_device_class": "hdd", 2020-03-11T22:43:43.485 INFO:tasks.workunit.client.0.smithi059.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4J2P7", 2020-03-11T22:43:43.485 INFO:tasks.workunit.client.0.smithi059.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:43:43.486 INFO:tasks.workunit.client.0.smithi059.stdout: "devices": "sda", 2020-03-11T22:43:43.486 INFO:tasks.workunit.client.0.smithi059.stdout: "distro": "centos", 2020-03-11T22:43:43.486 INFO:tasks.workunit.client.0.smithi059.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:43:43.486 INFO:tasks.workunit.client.0.smithi059.stdout: "distro_version": "8", 2020-03-11T22:43:43.486 INFO:tasks.workunit.client.0.smithi059.stdout: "front_addr": "[v2:127.0.0.1:6810/44413,v1:127.0.0.1:6811/44413]", 2020-03-11T22:43:43.486 INFO:tasks.workunit.client.0.smithi059.stdout: "front_iface": "", 2020-03-11T22:43:43.486 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/44413,v1:127.0.0.1:6817/44413]", 2020-03-11T22:43:43.487 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/44413,v1:127.0.0.1:6815/44413]", 2020-03-11T22:43:43.487 INFO:tasks.workunit.client.0.smithi059.stdout: "hostname": "smithi059", 2020-03-11T22:43:43.487 INFO:tasks.workunit.client.0.smithi059.stdout: "journal_rotational": "1", 2020-03-11T22:43:43.487 INFO:tasks.workunit.client.0.smithi059.stdout: "kernel_description": "#1 SMP Wed Feb 5 02:00:39 UTC 2020", 2020-03-11T22:43:43.487 INFO:tasks.workunit.client.0.smithi059.stdout: "kernel_version": "4.18.0-147.5.1.el8_1.x86_64", 2020-03-11T22:43:43.487 INFO:tasks.workunit.client.0.smithi059.stdout: "mem_swap_kb": "0", 2020-03-11T22:43:43.487 INFO:tasks.workunit.client.0.smithi059.stdout: "mem_total_kb": "32662084", 2020-03-11T22:43:43.488 INFO:tasks.workunit.client.0.smithi059.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:43:43.488 INFO:tasks.workunit.client.0.smithi059.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:43:43.488 INFO:tasks.workunit.client.0.smithi059.stdout: "os": "Linux", 2020-03-11T22:43:43.488 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_data": "td/divergent-priors/1", 2020-03-11T22:43:43.488 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:43:43.488 INFO:tasks.workunit.client.0.smithi059.stdout: "rotational": "1" 2020-03-11T22:43:43.489 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.489 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.489 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 2, 2020-03-11T22:43:43.489 INFO:tasks.workunit.client.0.smithi059.stdout: "arch": "x86_64", 2020-03-11T22:43:43.489 INFO:tasks.workunit.client.0.smithi059.stdout: "back_addr": "[v2:127.0.0.1:6812/51019,v1:127.0.0.1:6813/51019]", 2020-03-11T22:43:43.489 INFO:tasks.workunit.client.0.smithi059.stdout: "back_iface": "", 2020-03-11T22:43:43.489 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs": "1", 2020-03-11T22:43:43.490 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:43:43.490 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:43:43.490 INFO:tasks.workunit.client.0.smithi059.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:43:43.490 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:43:43.490 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:43:43.490 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:43:43.491 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:43:43.491 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2020-03-11T22:43:43.491 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:43:43.491 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:43:43.491 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:43:43.491 INFO:tasks.workunit.client.0.smithi059.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:43:43.491 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_release": "octopus", 2020-03-11T22:43:43.492 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:43:43.492 INFO:tasks.workunit.client.0.smithi059.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:43:43.492 INFO:tasks.workunit.client.0.smithi059.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-03-11T22:43:43.492 INFO:tasks.workunit.client.0.smithi059.stdout: "default_device_class": "hdd", 2020-03-11T22:43:43.492 INFO:tasks.workunit.client.0.smithi059.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4J2P7", 2020-03-11T22:43:43.492 INFO:tasks.workunit.client.0.smithi059.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:43:43.493 INFO:tasks.workunit.client.0.smithi059.stdout: "devices": "sda", 2020-03-11T22:43:43.493 INFO:tasks.workunit.client.0.smithi059.stdout: "distro": "centos", 2020-03-11T22:43:43.493 INFO:tasks.workunit.client.0.smithi059.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:43:43.493 INFO:tasks.workunit.client.0.smithi059.stdout: "distro_version": "8", 2020-03-11T22:43:43.493 INFO:tasks.workunit.client.0.smithi059.stdout: "front_addr": "[v2:127.0.0.1:6810/51019,v1:127.0.0.1:6811/51019]", 2020-03-11T22:43:43.493 INFO:tasks.workunit.client.0.smithi059.stdout: "front_iface": "", 2020-03-11T22:43:43.493 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/51019,v1:127.0.0.1:6817/51019]", 2020-03-11T22:43:43.494 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/51019,v1:127.0.0.1:6815/51019]", 2020-03-11T22:43:43.494 INFO:tasks.workunit.client.0.smithi059.stdout: "hostname": "smithi059", 2020-03-11T22:43:43.494 INFO:tasks.workunit.client.0.smithi059.stdout: "journal_rotational": "1", 2020-03-11T22:43:43.494 INFO:tasks.workunit.client.0.smithi059.stdout: "kernel_description": "#1 SMP Wed Feb 5 02:00:39 UTC 2020", 2020-03-11T22:43:43.494 INFO:tasks.workunit.client.0.smithi059.stdout: "kernel_version": "4.18.0-147.5.1.el8_1.x86_64", 2020-03-11T22:43:43.494 INFO:tasks.workunit.client.0.smithi059.stdout: "mem_swap_kb": "0", 2020-03-11T22:43:43.494 INFO:tasks.workunit.client.0.smithi059.stdout: "mem_total_kb": "32662084", 2020-03-11T22:43:43.494 INFO:tasks.workunit.client.0.smithi059.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:43:43.495 INFO:tasks.workunit.client.0.smithi059.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:43:43.495 INFO:tasks.workunit.client.0.smithi059.stdout: "os": "Linux", 2020-03-11T22:43:43.495 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_data": "td/divergent-priors/2", 2020-03-11T22:43:43.495 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:43:43.495 INFO:tasks.workunit.client.0.smithi059.stdout: "rotational": "1" 2020-03-11T22:43:43.495 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:43.496 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:43:43.496 INFO:tasks.workunit.client.0.smithi059.stdout: "osdmap_first_committed": 1, 2020-03-11T22:43:43.496 INFO:tasks.workunit.client.0.smithi059.stdout: "osdmap_last_committed": 45, 2020-03-11T22:43:43.496 INFO:tasks.workunit.client.0.smithi059.stdout: "crushmap": { 2020-03-11T22:43:43.496 INFO:tasks.workunit.client.0.smithi059.stdout: "devices": [ 2020-03-11T22:43:43.497 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.497 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 0, 2020-03-11T22:43:43.497 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "osd.0", 2020-03-11T22:43:43.497 INFO:tasks.workunit.client.0.smithi059.stdout: "class": "hdd" 2020-03-11T22:43:43.497 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.497 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.498 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 1, 2020-03-11T22:43:43.498 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "osd.1", 2020-03-11T22:43:43.498 INFO:tasks.workunit.client.0.smithi059.stdout: "class": "hdd" 2020-03-11T22:43:43.498 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.498 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.498 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 2, 2020-03-11T22:43:43.498 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "osd.2", 2020-03-11T22:43:43.499 INFO:tasks.workunit.client.0.smithi059.stdout: "class": "hdd" 2020-03-11T22:43:43.499 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:43.499 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:43:43.499 INFO:tasks.workunit.client.0.smithi059.stdout: "types": [ 2020-03-11T22:43:43.499 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.499 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 0, 2020-03-11T22:43:43.499 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "osd" 2020-03-11T22:43:43.500 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.500 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.500 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 1, 2020-03-11T22:43:43.500 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "host" 2020-03-11T22:43:43.500 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.500 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.501 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 2, 2020-03-11T22:43:43.501 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "chassis" 2020-03-11T22:43:43.501 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.501 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.501 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 3, 2020-03-11T22:43:43.501 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "rack" 2020-03-11T22:43:43.501 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.502 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.502 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 4, 2020-03-11T22:43:43.502 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "row" 2020-03-11T22:43:43.502 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.502 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.502 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 5, 2020-03-11T22:43:43.502 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "pdu" 2020-03-11T22:43:43.503 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.503 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.503 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 6, 2020-03-11T22:43:43.503 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "pod" 2020-03-11T22:43:43.503 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.503 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.503 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 7, 2020-03-11T22:43:43.504 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "room" 2020-03-11T22:43:43.504 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.504 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.504 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 8, 2020-03-11T22:43:43.504 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "datacenter" 2020-03-11T22:43:43.504 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.504 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.505 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 9, 2020-03-11T22:43:43.505 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "zone" 2020-03-11T22:43:43.505 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.505 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.505 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 10, 2020-03-11T22:43:43.505 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "region" 2020-03-11T22:43:43.505 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.506 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.506 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 11, 2020-03-11T22:43:43.506 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "root" 2020-03-11T22:43:43.506 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:43.506 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:43:43.506 INFO:tasks.workunit.client.0.smithi059.stdout: "buckets": [ 2020-03-11T22:43:43.506 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.507 INFO:tasks.workunit.client.0.smithi059.stdout: "id": -1, 2020-03-11T22:43:43.507 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "default", 2020-03-11T22:43:43.507 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 11, 2020-03-11T22:43:43.507 INFO:tasks.workunit.client.0.smithi059.stdout: "type_name": "root", 2020-03-11T22:43:43.507 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 19206, 2020-03-11T22:43:43.507 INFO:tasks.workunit.client.0.smithi059.stdout: "alg": "straw2", 2020-03-11T22:43:43.507 INFO:tasks.workunit.client.0.smithi059.stdout: "hash": "rjenkins1", 2020-03-11T22:43:43.507 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2020-03-11T22:43:43.508 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.508 INFO:tasks.workunit.client.0.smithi059.stdout: "id": -3, 2020-03-11T22:43:43.508 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 19206, 2020-03-11T22:43:43.508 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 0 2020-03-11T22:43:43.508 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:43.508 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:43:43.508 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.509 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.509 INFO:tasks.workunit.client.0.smithi059.stdout: "id": -2, 2020-03-11T22:43:43.509 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "default~hdd", 2020-03-11T22:43:43.509 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 11, 2020-03-11T22:43:43.509 INFO:tasks.workunit.client.0.smithi059.stdout: "type_name": "root", 2020-03-11T22:43:43.509 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 19206, 2020-03-11T22:43:43.509 INFO:tasks.workunit.client.0.smithi059.stdout: "alg": "straw2", 2020-03-11T22:43:43.510 INFO:tasks.workunit.client.0.smithi059.stdout: "hash": "rjenkins1", 2020-03-11T22:43:43.510 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2020-03-11T22:43:43.510 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.510 INFO:tasks.workunit.client.0.smithi059.stdout: "id": -4, 2020-03-11T22:43:43.510 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 19206, 2020-03-11T22:43:43.510 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 0 2020-03-11T22:43:43.510 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:43.510 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:43:43.511 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.511 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.511 INFO:tasks.workunit.client.0.smithi059.stdout: "id": -3, 2020-03-11T22:43:43.511 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "smithi059", 2020-03-11T22:43:43.511 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 1, 2020-03-11T22:43:43.511 INFO:tasks.workunit.client.0.smithi059.stdout: "type_name": "host", 2020-03-11T22:43:43.511 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 19206, 2020-03-11T22:43:43.512 INFO:tasks.workunit.client.0.smithi059.stdout: "alg": "straw2", 2020-03-11T22:43:43.512 INFO:tasks.workunit.client.0.smithi059.stdout: "hash": "rjenkins1", 2020-03-11T22:43:43.512 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2020-03-11T22:43:43.512 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.512 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 0, 2020-03-11T22:43:43.512 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 6402, 2020-03-11T22:43:43.512 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 0 2020-03-11T22:43:43.513 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.513 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.513 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 1, 2020-03-11T22:43:43.513 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 6402, 2020-03-11T22:43:43.513 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 1 2020-03-11T22:43:43.513 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.513 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.513 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 2, 2020-03-11T22:43:43.514 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 6402, 2020-03-11T22:43:43.514 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 2 2020-03-11T22:43:43.514 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:43.514 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:43:43.514 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.514 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.514 INFO:tasks.workunit.client.0.smithi059.stdout: "id": -4, 2020-03-11T22:43:43.515 INFO:tasks.workunit.client.0.smithi059.stdout: "name": "smithi059~hdd", 2020-03-11T22:43:43.515 INFO:tasks.workunit.client.0.smithi059.stdout: "type_id": 1, 2020-03-11T22:43:43.515 INFO:tasks.workunit.client.0.smithi059.stdout: "type_name": "host", 2020-03-11T22:43:43.515 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 19206, 2020-03-11T22:43:43.515 INFO:tasks.workunit.client.0.smithi059.stdout: "alg": "straw2", 2020-03-11T22:43:43.515 INFO:tasks.workunit.client.0.smithi059.stdout: "hash": "rjenkins1", 2020-03-11T22:43:43.515 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2020-03-11T22:43:43.516 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.516 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 0, 2020-03-11T22:43:43.516 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 6402, 2020-03-11T22:43:43.516 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 0 2020-03-11T22:43:43.516 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.516 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.516 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 1, 2020-03-11T22:43:43.516 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 6402, 2020-03-11T22:43:43.517 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 1 2020-03-11T22:43:43.517 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.517 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.517 INFO:tasks.workunit.client.0.smithi059.stdout: "id": 2, 2020-03-11T22:43:43.517 INFO:tasks.workunit.client.0.smithi059.stdout: "weight": 6402, 2020-03-11T22:43:43.517 INFO:tasks.workunit.client.0.smithi059.stdout: "pos": 2 2020-03-11T22:43:43.517 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:43.518 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:43:43.518 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:43.518 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:43:43.518 INFO:tasks.workunit.client.0.smithi059.stdout: "rules": [ 2020-03-11T22:43:43.518 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.518 INFO:tasks.workunit.client.0.smithi059.stdout: "rule_id": 0, 2020-03-11T22:43:43.518 INFO:tasks.workunit.client.0.smithi059.stdout: "rule_name": "replicated_rule", 2020-03-11T22:43:43.519 INFO:tasks.workunit.client.0.smithi059.stdout: "ruleset": 0, 2020-03-11T22:43:43.519 INFO:tasks.workunit.client.0.smithi059.stdout: "type": 1, 2020-03-11T22:43:43.519 INFO:tasks.workunit.client.0.smithi059.stdout: "min_size": 1, 2020-03-11T22:43:43.519 INFO:tasks.workunit.client.0.smithi059.stdout: "max_size": 10, 2020-03-11T22:43:43.519 INFO:tasks.workunit.client.0.smithi059.stdout: "steps": [ 2020-03-11T22:43:43.519 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.519 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "take", 2020-03-11T22:43:43.519 INFO:tasks.workunit.client.0.smithi059.stdout: "item": -1, 2020-03-11T22:43:43.520 INFO:tasks.workunit.client.0.smithi059.stdout: "item_name": "default" 2020-03-11T22:43:43.520 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.520 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.520 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "choose_firstn", 2020-03-11T22:43:43.520 INFO:tasks.workunit.client.0.smithi059.stdout: "num": 0, 2020-03-11T22:43:43.520 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "osd" 2020-03-11T22:43:43.520 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.521 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.522 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "emit" 2020-03-11T22:43:43.522 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:43.522 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:43:43.522 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.522 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.522 INFO:tasks.workunit.client.0.smithi059.stdout: "rule_id": 1, 2020-03-11T22:43:43.523 INFO:tasks.workunit.client.0.smithi059.stdout: "rule_name": "test", 2020-03-11T22:43:43.523 INFO:tasks.workunit.client.0.smithi059.stdout: "ruleset": 1, 2020-03-11T22:43:43.523 INFO:tasks.workunit.client.0.smithi059.stdout: "type": 3, 2020-03-11T22:43:43.523 INFO:tasks.workunit.client.0.smithi059.stdout: "min_size": 3, 2020-03-11T22:43:43.523 INFO:tasks.workunit.client.0.smithi059.stdout: "max_size": 3, 2020-03-11T22:43:43.523 INFO:tasks.workunit.client.0.smithi059.stdout: "steps": [ 2020-03-11T22:43:43.523 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.524 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "set_chooseleaf_tries", 2020-03-11T22:43:43.524 INFO:tasks.workunit.client.0.smithi059.stdout: "num": 5 2020-03-11T22:43:43.524 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.524 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.524 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "set_choose_tries", 2020-03-11T22:43:43.524 INFO:tasks.workunit.client.0.smithi059.stdout: "num": 100 2020-03-11T22:43:43.524 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.525 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.525 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "take", 2020-03-11T22:43:43.525 INFO:tasks.workunit.client.0.smithi059.stdout: "item": -1, 2020-03-11T22:43:43.525 INFO:tasks.workunit.client.0.smithi059.stdout: "item_name": "default" 2020-03-11T22:43:43.525 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.525 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.525 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "choose_indep", 2020-03-11T22:43:43.526 INFO:tasks.workunit.client.0.smithi059.stdout: "num": 0, 2020-03-11T22:43:43.526 INFO:tasks.workunit.client.0.smithi059.stdout: "type": "osd" 2020-03-11T22:43:43.526 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.526 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.526 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "emit" 2020-03-11T22:43:43.526 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:43.526 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:43:43.527 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:43.527 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:43:43.527 INFO:tasks.workunit.client.0.smithi059.stdout: "tunables": { 2020-03-11T22:43:43.527 INFO:tasks.workunit.client.0.smithi059.stdout: "choose_local_tries": 0, 2020-03-11T22:43:43.527 INFO:tasks.workunit.client.0.smithi059.stdout: "choose_local_fallback_tries": 0, 2020-03-11T22:43:43.527 INFO:tasks.workunit.client.0.smithi059.stdout: "choose_total_tries": 50, 2020-03-11T22:43:43.527 INFO:tasks.workunit.client.0.smithi059.stdout: "chooseleaf_descend_once": 1, 2020-03-11T22:43:43.528 INFO:tasks.workunit.client.0.smithi059.stdout: "chooseleaf_vary_r": 1, 2020-03-11T22:43:43.528 INFO:tasks.workunit.client.0.smithi059.stdout: "chooseleaf_stable": 1, 2020-03-11T22:43:43.528 INFO:tasks.workunit.client.0.smithi059.stdout: "straw_calc_version": 1, 2020-03-11T22:43:43.528 INFO:tasks.workunit.client.0.smithi059.stdout: "allowed_bucket_algs": 54, 2020-03-11T22:43:43.528 INFO:tasks.workunit.client.0.smithi059.stdout: "profile": "jewel", 2020-03-11T22:43:43.528 INFO:tasks.workunit.client.0.smithi059.stdout: "optimal_tunables": 1, 2020-03-11T22:43:43.528 INFO:tasks.workunit.client.0.smithi059.stdout: "legacy_tunables": 0, 2020-03-11T22:43:43.529 INFO:tasks.workunit.client.0.smithi059.stdout: "minimum_required_version": "jewel", 2020-03-11T22:43:43.529 INFO:tasks.workunit.client.0.smithi059.stdout: "require_feature_tunables": 1, 2020-03-11T22:43:43.529 INFO:tasks.workunit.client.0.smithi059.stdout: "require_feature_tunables2": 1, 2020-03-11T22:43:43.529 INFO:tasks.workunit.client.0.smithi059.stdout: "has_v2_rules": 1, 2020-03-11T22:43:43.529 INFO:tasks.workunit.client.0.smithi059.stdout: "require_feature_tunables3": 1, 2020-03-11T22:43:43.529 INFO:tasks.workunit.client.0.smithi059.stdout: "has_v3_rules": 0, 2020-03-11T22:43:43.529 INFO:tasks.workunit.client.0.smithi059.stdout: "has_v4_buckets": 1, 2020-03-11T22:43:43.530 INFO:tasks.workunit.client.0.smithi059.stdout: "require_feature_tunables5": 1, 2020-03-11T22:43:43.530 INFO:tasks.workunit.client.0.smithi059.stdout: "has_v5_rules": 0 2020-03-11T22:43:43.530 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.530 INFO:tasks.workunit.client.0.smithi059.stdout: "choose_args": {} 2020-03-11T22:43:43.530 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.530 INFO:tasks.workunit.client.0.smithi059.stdout: "fsmap": { 2020-03-11T22:43:43.530 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": 1, 2020-03-11T22:43:43.531 INFO:tasks.workunit.client.0.smithi059.stdout: "default_fscid": -1, 2020-03-11T22:43:43.531 INFO:tasks.workunit.client.0.smithi059.stdout: "compat": { 2020-03-11T22:43:43.531 INFO:tasks.workunit.client.0.smithi059.stdout: "compat": {}, 2020-03-11T22:43:43.531 INFO:tasks.workunit.client.0.smithi059.stdout: "ro_compat": {}, 2020-03-11T22:43:43.531 INFO:tasks.workunit.client.0.smithi059.stdout: "incompat": { 2020-03-11T22:43:43.531 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_1": "base v0.20", 2020-03-11T22:43:43.531 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_2": "client writeable ranges", 2020-03-11T22:43:43.532 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_3": "default file layouts on dirs", 2020-03-11T22:43:43.532 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_4": "dir inode in separate object", 2020-03-11T22:43:43.532 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_5": "mds uses versioned encoding", 2020-03-11T22:43:43.532 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_6": "dirfrag is stored in omap", 2020-03-11T22:43:43.532 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_8": "no anchor table", 2020-03-11T22:43:43.532 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_9": "file layout v2", 2020-03-11T22:43:43.532 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_10": "snaprealm v2" 2020-03-11T22:43:43.533 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:43.533 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.533 INFO:tasks.workunit.client.0.smithi059.stdout: "feature_flags": { 2020-03-11T22:43:43.533 INFO:tasks.workunit.client.0.smithi059.stdout: "enable_multiple": false, 2020-03-11T22:43:43.533 INFO:tasks.workunit.client.0.smithi059.stdout: "ever_enabled_multiple": false 2020-03-11T22:43:43.533 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.534 INFO:tasks.workunit.client.0.smithi059.stdout: "standbys": [], 2020-03-11T22:43:43.534 INFO:tasks.workunit.client.0.smithi059.stdout: "filesystems": [] 2020-03-11T22:43:43.534 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.534 INFO:tasks.workunit.client.0.smithi059.stdout: "mdsmap_first_committed": 1, 2020-03-11T22:43:43.534 INFO:tasks.workunit.client.0.smithi059.stdout: "mdsmap_last_committed": 1, 2020-03-11T22:43:43.534 INFO:tasks.workunit.client.0.smithi059.stdout: "auth": { 2020-03-11T22:43:43.534 INFO:tasks.workunit.client.0.smithi059.stdout: "first_committed": 1, 2020-03-11T22:43:43.535 INFO:tasks.workunit.client.0.smithi059.stdout: "last_committed": 5, 2020-03-11T22:43:43.535 INFO:tasks.workunit.client.0.smithi059.stdout: "num_secrets": 3 2020-03-11T22:43:43.535 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.535 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pg": 1, 2020-03-11T22:43:43.535 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pg_active": 1, 2020-03-11T22:43:43.535 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pg_unknown": 0, 2020-03-11T22:43:43.535 INFO:tasks.workunit.client.0.smithi059.stdout: "num_osd": 3, 2020-03-11T22:43:43.535 INFO:tasks.workunit.client.0.smithi059.stdout: "pool_sum": { 2020-03-11T22:43:43.536 INFO:tasks.workunit.client.0.smithi059.stdout: "stat_sum": { 2020-03-11T22:43:43.536 INFO:tasks.workunit.client.0.smithi059.stdout: "num_bytes": 48430, 2020-03-11T22:43:43.536 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects": 101, 2020-03-11T22:43:43.536 INFO:tasks.workunit.client.0.smithi059.stdout: "num_object_clones": 1, 2020-03-11T22:43:43.536 INFO:tasks.workunit.client.0.smithi059.stdout: "num_object_copies": 303, 2020-03-11T22:43:43.536 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:43:43.536 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_missing": 0, 2020-03-11T22:43:43.537 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_degraded": 101, 2020-03-11T22:43:43.537 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_misplaced": 0, 2020-03-11T22:43:43.537 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_unfound": 0, 2020-03-11T22:43:43.537 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_dirty": 101, 2020-03-11T22:43:43.537 INFO:tasks.workunit.client.0.smithi059.stdout: "num_whiteouts": 0, 2020-03-11T22:43:43.537 INFO:tasks.workunit.client.0.smithi059.stdout: "num_read": 0, 2020-03-11T22:43:43.537 INFO:tasks.workunit.client.0.smithi059.stdout: "num_read_kb": 0, 2020-03-11T22:43:43.538 INFO:tasks.workunit.client.0.smithi059.stdout: "num_write": 101, 2020-03-11T22:43:43.538 INFO:tasks.workunit.client.0.smithi059.stdout: "num_write_kb": 101, 2020-03-11T22:43:43.538 INFO:tasks.workunit.client.0.smithi059.stdout: "num_scrub_errors": 0, 2020-03-11T22:43:43.538 INFO:tasks.workunit.client.0.smithi059.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:43:43.538 INFO:tasks.workunit.client.0.smithi059.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:43:43.538 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_recovered": 0, 2020-03-11T22:43:43.539 INFO:tasks.workunit.client.0.smithi059.stdout: "num_bytes_recovered": 0, 2020-03-11T22:43:43.539 INFO:tasks.workunit.client.0.smithi059.stdout: "num_keys_recovered": 0, 2020-03-11T22:43:43.539 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_omap": 0, 2020-03-11T22:43:43.539 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:43:43.539 INFO:tasks.workunit.client.0.smithi059.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:43:43.539 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush": 0, 2020-03-11T22:43:43.539 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush_kb": 0, 2020-03-11T22:43:43.540 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict": 0, 2020-03-11T22:43:43.540 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict_kb": 0, 2020-03-11T22:43:43.540 INFO:tasks.workunit.client.0.smithi059.stdout: "num_promote": 0, 2020-03-11T22:43:43.540 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush_mode_high": 0, 2020-03-11T22:43:43.540 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush_mode_low": 0, 2020-03-11T22:43:43.540 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict_mode_some": 0, 2020-03-11T22:43:43.540 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict_mode_full": 0, 2020-03-11T22:43:43.541 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_pinned": 0, 2020-03-11T22:43:43.541 INFO:tasks.workunit.client.0.smithi059.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:43:43.541 INFO:tasks.workunit.client.0.smithi059.stdout: "num_large_omap_objects": 0, 2020-03-11T22:43:43.541 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_manifest": 0, 2020-03-11T22:43:43.541 INFO:tasks.workunit.client.0.smithi059.stdout: "num_omap_bytes": 0, 2020-03-11T22:43:43.541 INFO:tasks.workunit.client.0.smithi059.stdout: "num_omap_keys": 0, 2020-03-11T22:43:43.541 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_repaired": 0 2020-03-11T22:43:43.542 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.542 INFO:tasks.workunit.client.0.smithi059.stdout: "store_stats": { 2020-03-11T22:43:43.542 INFO:tasks.workunit.client.0.smithi059.stdout: "total": 0, 2020-03-11T22:43:43.542 INFO:tasks.workunit.client.0.smithi059.stdout: "available": 0, 2020-03-11T22:43:43.542 INFO:tasks.workunit.client.0.smithi059.stdout: "internally_reserved": 0, 2020-03-11T22:43:43.542 INFO:tasks.workunit.client.0.smithi059.stdout: "allocated": 0, 2020-03-11T22:43:43.542 INFO:tasks.workunit.client.0.smithi059.stdout: "data_stored": 0, 2020-03-11T22:43:43.543 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed": 0, 2020-03-11T22:43:43.543 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_allocated": 0, 2020-03-11T22:43:43.543 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_original": 0, 2020-03-11T22:43:43.543 INFO:tasks.workunit.client.0.smithi059.stdout: "omap_allocated": 0, 2020-03-11T22:43:43.543 INFO:tasks.workunit.client.0.smithi059.stdout: "internal_metadata": 0 2020-03-11T22:43:43.543 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.543 INFO:tasks.workunit.client.0.smithi059.stdout: "log_size": 18, 2020-03-11T22:43:43.544 INFO:tasks.workunit.client.0.smithi059.stdout: "ondisk_log_size": 18, 2020-03-11T22:43:43.544 INFO:tasks.workunit.client.0.smithi059.stdout: "up": 3, 2020-03-11T22:43:43.544 INFO:tasks.workunit.client.0.smithi059.stdout: "acting": 3, 2020-03-11T22:43:43.544 INFO:tasks.workunit.client.0.smithi059.stdout: "num_store_stats": 0 2020-03-11T22:43:43.544 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.544 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_sum": { 2020-03-11T22:43:43.544 INFO:tasks.workunit.client.0.smithi059.stdout: "up_from": 0, 2020-03-11T22:43:43.545 INFO:tasks.workunit.client.0.smithi059.stdout: "seq": 0, 2020-03-11T22:43:43.545 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pgs": 3, 2020-03-11T22:43:43.545 INFO:tasks.workunit.client.0.smithi059.stdout: "num_osds": 3, 2020-03-11T22:43:43.545 INFO:tasks.workunit.client.0.smithi059.stdout: "num_per_pool_osds": 3, 2020-03-11T22:43:43.545 INFO:tasks.workunit.client.0.smithi059.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:43:43.545 INFO:tasks.workunit.client.0.smithi059.stdout: "kb": 314572800, 2020-03-11T22:43:43.545 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used": 3181056, 2020-03-11T22:43:43.545 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used_data": 35008, 2020-03-11T22:43:43.546 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used_omap": 9, 2020-03-11T22:43:43.546 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used_meta": 3145718, 2020-03-11T22:43:43.546 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_avail": 311391744, 2020-03-11T22:43:43.546 INFO:tasks.workunit.client.0.smithi059.stdout: "statfs": { 2020-03-11T22:43:43.546 INFO:tasks.workunit.client.0.smithi059.stdout: "total": 322122547200, 2020-03-11T22:43:43.546 INFO:tasks.workunit.client.0.smithi059.stdout: "available": 318865145856, 2020-03-11T22:43:43.546 INFO:tasks.workunit.client.0.smithi059.stdout: "internally_reserved": 3221225472, 2020-03-11T22:43:43.547 INFO:tasks.workunit.client.0.smithi059.stdout: "allocated": 35848192, 2020-03-11T22:43:43.547 INFO:tasks.workunit.client.0.smithi059.stdout: "data_stored": 1556411, 2020-03-11T22:43:43.547 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed": 0, 2020-03-11T22:43:43.547 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_allocated": 0, 2020-03-11T22:43:43.547 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_original": 0, 2020-03-11T22:43:43.547 INFO:tasks.workunit.client.0.smithi059.stdout: "omap_allocated": 9486, 2020-03-11T22:43:43.547 INFO:tasks.workunit.client.0.smithi059.stdout: "internal_metadata": 3221215986 2020-03-11T22:43:43.548 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.548 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_peers": [], 2020-03-11T22:43:43.548 INFO:tasks.workunit.client.0.smithi059.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:43:43.548 INFO:tasks.workunit.client.0.smithi059.stdout: "num_snap_trimming": 0, 2020-03-11T22:43:43.548 INFO:tasks.workunit.client.0.smithi059.stdout: "num_shards_repaired": 0, 2020-03-11T22:43:43.548 INFO:tasks.workunit.client.0.smithi059.stdout: "op_queue_age_hist": { 2020-03-11T22:43:43.548 INFO:tasks.workunit.client.0.smithi059.stdout: "histogram": [], 2020-03-11T22:43:43.549 INFO:tasks.workunit.client.0.smithi059.stdout: "upper_bound": 1 2020-03-11T22:43:43.549 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.549 INFO:tasks.workunit.client.0.smithi059.stdout: "perf_stat": { 2020-03-11T22:43:43.549 INFO:tasks.workunit.client.0.smithi059.stdout: "commit_latency_ms": 0, 2020-03-11T22:43:43.549 INFO:tasks.workunit.client.0.smithi059.stdout: "apply_latency_ms": 0, 2020-03-11T22:43:43.549 INFO:tasks.workunit.client.0.smithi059.stdout: "commit_latency_ns": 0, 2020-03-11T22:43:43.549 INFO:tasks.workunit.client.0.smithi059.stdout: "apply_latency_ns": 0 2020-03-11T22:43:43.550 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.550 INFO:tasks.workunit.client.0.smithi059.stdout: "alerts": [], 2020-03-11T22:43:43.550 INFO:tasks.workunit.client.0.smithi059.stdout: "network_ping_times": [] 2020-03-11T22:43:43.550 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.550 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_sum_by_class": { 2020-03-11T22:43:43.550 INFO:tasks.workunit.client.0.smithi059.stdout: "hdd": { 2020-03-11T22:43:43.550 INFO:tasks.workunit.client.0.smithi059.stdout: "up_from": 0, 2020-03-11T22:43:43.551 INFO:tasks.workunit.client.0.smithi059.stdout: "seq": 0, 2020-03-11T22:43:43.551 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pgs": 3, 2020-03-11T22:43:43.551 INFO:tasks.workunit.client.0.smithi059.stdout: "num_osds": 3, 2020-03-11T22:43:43.551 INFO:tasks.workunit.client.0.smithi059.stdout: "num_per_pool_osds": 3, 2020-03-11T22:43:43.551 INFO:tasks.workunit.client.0.smithi059.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:43:43.551 INFO:tasks.workunit.client.0.smithi059.stdout: "kb": 314572800, 2020-03-11T22:43:43.551 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used": 3181056, 2020-03-11T22:43:43.552 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used_data": 35008, 2020-03-11T22:43:43.552 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used_omap": 9, 2020-03-11T22:43:43.552 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_used_meta": 3145718, 2020-03-11T22:43:43.552 INFO:tasks.workunit.client.0.smithi059.stdout: "kb_avail": 311391744, 2020-03-11T22:43:43.552 INFO:tasks.workunit.client.0.smithi059.stdout: "statfs": { 2020-03-11T22:43:43.552 INFO:tasks.workunit.client.0.smithi059.stdout: "total": 322122547200, 2020-03-11T22:43:43.552 INFO:tasks.workunit.client.0.smithi059.stdout: "available": 318865145856, 2020-03-11T22:43:43.552 INFO:tasks.workunit.client.0.smithi059.stdout: "internally_reserved": 3221225472, 2020-03-11T22:43:43.553 INFO:tasks.workunit.client.0.smithi059.stdout: "allocated": 35848192, 2020-03-11T22:43:43.553 INFO:tasks.workunit.client.0.smithi059.stdout: "data_stored": 1556411, 2020-03-11T22:43:43.553 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed": 0, 2020-03-11T22:43:43.553 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_allocated": 0, 2020-03-11T22:43:43.553 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_original": 0, 2020-03-11T22:43:43.553 INFO:tasks.workunit.client.0.smithi059.stdout: "omap_allocated": 9486, 2020-03-11T22:43:43.553 INFO:tasks.workunit.client.0.smithi059.stdout: "internal_metadata": 3221215986 2020-03-11T22:43:43.554 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.554 INFO:tasks.workunit.client.0.smithi059.stdout: "hb_peers": [], 2020-03-11T22:43:43.554 INFO:tasks.workunit.client.0.smithi059.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:43:43.554 INFO:tasks.workunit.client.0.smithi059.stdout: "num_snap_trimming": 0, 2020-03-11T22:43:43.554 INFO:tasks.workunit.client.0.smithi059.stdout: "num_shards_repaired": 0, 2020-03-11T22:43:43.554 INFO:tasks.workunit.client.0.smithi059.stdout: "op_queue_age_hist": { 2020-03-11T22:43:43.554 INFO:tasks.workunit.client.0.smithi059.stdout: "histogram": [], 2020-03-11T22:43:43.555 INFO:tasks.workunit.client.0.smithi059.stdout: "upper_bound": 1 2020-03-11T22:43:43.555 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.555 INFO:tasks.workunit.client.0.smithi059.stdout: "perf_stat": { 2020-03-11T22:43:43.555 INFO:tasks.workunit.client.0.smithi059.stdout: "commit_latency_ms": 0, 2020-03-11T22:43:43.555 INFO:tasks.workunit.client.0.smithi059.stdout: "apply_latency_ms": 0, 2020-03-11T22:43:43.555 INFO:tasks.workunit.client.0.smithi059.stdout: "commit_latency_ns": 0, 2020-03-11T22:43:43.555 INFO:tasks.workunit.client.0.smithi059.stdout: "apply_latency_ns": 0 2020-03-11T22:43:43.556 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.556 INFO:tasks.workunit.client.0.smithi059.stdout: "alerts": [], 2020-03-11T22:43:43.556 INFO:tasks.workunit.client.0.smithi059.stdout: "network_ping_times": [] 2020-03-11T22:43:43.556 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:43.556 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.556 INFO:tasks.workunit.client.0.smithi059.stdout: "pool_stats": [ 2020-03-11T22:43:43.556 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.557 INFO:tasks.workunit.client.0.smithi059.stdout: "poolid": 1, 2020-03-11T22:43:43.557 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pg": 1, 2020-03-11T22:43:43.557 INFO:tasks.workunit.client.0.smithi059.stdout: "stat_sum": { 2020-03-11T22:43:43.557 INFO:tasks.workunit.client.0.smithi059.stdout: "num_bytes": 48430, 2020-03-11T22:43:43.557 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects": 101, 2020-03-11T22:43:43.557 INFO:tasks.workunit.client.0.smithi059.stdout: "num_object_clones": 1, 2020-03-11T22:43:43.558 INFO:tasks.workunit.client.0.smithi059.stdout: "num_object_copies": 303, 2020-03-11T22:43:43.558 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:43:43.558 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_missing": 0, 2020-03-11T22:43:43.558 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_degraded": 101, 2020-03-11T22:43:43.558 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_misplaced": 0, 2020-03-11T22:43:43.558 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_unfound": 0, 2020-03-11T22:43:43.558 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_dirty": 101, 2020-03-11T22:43:43.559 INFO:tasks.workunit.client.0.smithi059.stdout: "num_whiteouts": 0, 2020-03-11T22:43:43.559 INFO:tasks.workunit.client.0.smithi059.stdout: "num_read": 0, 2020-03-11T22:43:43.559 INFO:tasks.workunit.client.0.smithi059.stdout: "num_read_kb": 0, 2020-03-11T22:43:43.559 INFO:tasks.workunit.client.0.smithi059.stdout: "num_write": 101, 2020-03-11T22:43:43.559 INFO:tasks.workunit.client.0.smithi059.stdout: "num_write_kb": 101, 2020-03-11T22:43:43.559 INFO:tasks.workunit.client.0.smithi059.stdout: "num_scrub_errors": 0, 2020-03-11T22:43:43.559 INFO:tasks.workunit.client.0.smithi059.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:43:43.560 INFO:tasks.workunit.client.0.smithi059.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:43:43.560 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_recovered": 0, 2020-03-11T22:43:43.560 INFO:tasks.workunit.client.0.smithi059.stdout: "num_bytes_recovered": 0, 2020-03-11T22:43:43.560 INFO:tasks.workunit.client.0.smithi059.stdout: "num_keys_recovered": 0, 2020-03-11T22:43:43.560 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_omap": 0, 2020-03-11T22:43:43.560 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:43:43.560 INFO:tasks.workunit.client.0.smithi059.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:43:43.561 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush": 0, 2020-03-11T22:43:43.561 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush_kb": 0, 2020-03-11T22:43:43.561 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict": 0, 2020-03-11T22:43:43.561 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict_kb": 0, 2020-03-11T22:43:43.561 INFO:tasks.workunit.client.0.smithi059.stdout: "num_promote": 0, 2020-03-11T22:43:43.561 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush_mode_high": 0, 2020-03-11T22:43:43.561 INFO:tasks.workunit.client.0.smithi059.stdout: "num_flush_mode_low": 0, 2020-03-11T22:43:43.562 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict_mode_some": 0, 2020-03-11T22:43:43.562 INFO:tasks.workunit.client.0.smithi059.stdout: "num_evict_mode_full": 0, 2020-03-11T22:43:43.562 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_pinned": 0, 2020-03-11T22:43:43.562 INFO:tasks.workunit.client.0.smithi059.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:43:43.562 INFO:tasks.workunit.client.0.smithi059.stdout: "num_large_omap_objects": 0, 2020-03-11T22:43:43.562 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_manifest": 0, 2020-03-11T22:43:43.562 INFO:tasks.workunit.client.0.smithi059.stdout: "num_omap_bytes": 0, 2020-03-11T22:43:43.563 INFO:tasks.workunit.client.0.smithi059.stdout: "num_omap_keys": 0, 2020-03-11T22:43:43.563 INFO:tasks.workunit.client.0.smithi059.stdout: "num_objects_repaired": 0 2020-03-11T22:43:43.563 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.563 INFO:tasks.workunit.client.0.smithi059.stdout: "store_stats": { 2020-03-11T22:43:43.563 INFO:tasks.workunit.client.0.smithi059.stdout: "total": 0, 2020-03-11T22:43:43.563 INFO:tasks.workunit.client.0.smithi059.stdout: "available": 0, 2020-03-11T22:43:43.563 INFO:tasks.workunit.client.0.smithi059.stdout: "internally_reserved": 0, 2020-03-11T22:43:43.564 INFO:tasks.workunit.client.0.smithi059.stdout: "allocated": 19857408, 2020-03-11T22:43:43.564 INFO:tasks.workunit.client.0.smithi059.stdout: "data_stored": 1241088, 2020-03-11T22:43:43.564 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed": 0, 2020-03-11T22:43:43.564 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_allocated": 0, 2020-03-11T22:43:43.564 INFO:tasks.workunit.client.0.smithi059.stdout: "data_compressed_original": 0, 2020-03-11T22:43:43.564 INFO:tasks.workunit.client.0.smithi059.stdout: "omap_allocated": 0, 2020-03-11T22:43:43.564 INFO:tasks.workunit.client.0.smithi059.stdout: "internal_metadata": 0 2020-03-11T22:43:43.565 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.565 INFO:tasks.workunit.client.0.smithi059.stdout: "log_size": 18, 2020-03-11T22:43:43.565 INFO:tasks.workunit.client.0.smithi059.stdout: "ondisk_log_size": 18, 2020-03-11T22:43:43.565 INFO:tasks.workunit.client.0.smithi059.stdout: "up": 3, 2020-03-11T22:43:43.565 INFO:tasks.workunit.client.0.smithi059.stdout: "acting": 3, 2020-03-11T22:43:43.565 INFO:tasks.workunit.client.0.smithi059.stdout: "num_store_stats": 3 2020-03-11T22:43:43.565 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:43.566 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:43:43.566 INFO:tasks.workunit.client.0.smithi059.stdout: "osd_stats": [ 2020-03-11T22:43:43.566 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.566 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 0, 2020-03-11T22:43:43.566 INFO:tasks.workunit.client.0.smithi059.stdout: "seq": 176093659146 2020-03-11T22:43:43.566 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.566 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.567 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 1, 2020-03-11T22:43:43.567 INFO:tasks.workunit.client.0.smithi059.stdout: "seq": 42949672978 2020-03-11T22:43:43.567 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.567 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.567 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 2, 2020-03-11T22:43:43.567 INFO:tasks.workunit.client.0.smithi059.stdout: "seq": 188978561034 2020-03-11T22:43:43.567 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:43.568 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:43:43.568 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pg_by_state": [ 2020-03-11T22:43:43.568 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.568 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "active+undersized+degraded", 2020-03-11T22:43:43.568 INFO:tasks.workunit.client.0.smithi059.stdout: "num": 1 2020-03-11T22:43:43.568 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:43.568 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:43:43.568 INFO:tasks.workunit.client.0.smithi059.stdout: "num_pg_by_osd": [ 2020-03-11T22:43:43.569 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.569 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 2, 2020-03-11T22:43:43.569 INFO:tasks.workunit.client.0.smithi059.stdout: "num_primary_pg": 0, 2020-03-11T22:43:43.569 INFO:tasks.workunit.client.0.smithi059.stdout: "num_acting_pg": 1, 2020-03-11T22:43:43.569 INFO:tasks.workunit.client.0.smithi059.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:43:43.569 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.569 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.570 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 2147483647, 2020-03-11T22:43:43.570 INFO:tasks.workunit.client.0.smithi059.stdout: "num_primary_pg": 0, 2020-03-11T22:43:43.570 INFO:tasks.workunit.client.0.smithi059.stdout: "num_acting_pg": 1, 2020-03-11T22:43:43.570 INFO:tasks.workunit.client.0.smithi059.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:43:43.570 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.570 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.570 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 1, 2020-03-11T22:43:43.571 INFO:tasks.workunit.client.0.smithi059.stdout: "num_primary_pg": 0, 2020-03-11T22:43:43.571 INFO:tasks.workunit.client.0.smithi059.stdout: "num_acting_pg": 0, 2020-03-11T22:43:43.571 INFO:tasks.workunit.client.0.smithi059.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:43:43.571 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.571 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.571 INFO:tasks.workunit.client.0.smithi059.stdout: "osd": 0, 2020-03-11T22:43:43.571 INFO:tasks.workunit.client.0.smithi059.stdout: "num_primary_pg": 1, 2020-03-11T22:43:43.572 INFO:tasks.workunit.client.0.smithi059.stdout: "num_acting_pg": 1, 2020-03-11T22:43:43.572 INFO:tasks.workunit.client.0.smithi059.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:43:43.572 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:43.572 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:43:43.572 INFO:tasks.workunit.client.0.smithi059.stdout: "purged_snaps": [ 2020-03-11T22:43:43.572 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:43:43.572 INFO:tasks.workunit.client.0.smithi059.stdout: "pool": 1, 2020-03-11T22:43:43.573 INFO:tasks.workunit.client.0.smithi059.stdout: "purged_snaps": {} 2020-03-11T22:43:43.573 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:43.573 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:43:43.573 INFO:tasks.workunit.client.0.smithi059.stdout: "servicemap": { 2020-03-11T22:43:43.573 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": 1, 2020-03-11T22:43:43.573 INFO:tasks.workunit.client.0.smithi059.stdout: "modified": "2020-03-11T22:41:10.840801+0000", 2020-03-11T22:43:43.573 INFO:tasks.workunit.client.0.smithi059.stdout: "services": {} 2020-03-11T22:43:43.573 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:43:43.574 INFO:tasks.workunit.client.0.smithi059.stdout: "paxos": { 2020-03-11T22:43:43.574 INFO:tasks.workunit.client.0.smithi059.stdout: "first_committed": 1, 2020-03-11T22:43:43.574 INFO:tasks.workunit.client.0.smithi059.stdout: "last_committed": 433, 2020-03-11T22:43:43.574 INFO:tasks.workunit.client.0.smithi059.stdout: "last_pn": 0, 2020-03-11T22:43:43.574 INFO:tasks.workunit.client.0.smithi059.stdout: "accepted_pn": 0 2020-03-11T22:43:43.574 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:43:43.574 INFO:tasks.workunit.client.0.smithi059.stdout:} 2020-03-11T22:43:43.575 INFO:tasks.workunit.client.0.smithi059.stderr:report 2484012031 2020-03-11T22:43:43.576 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: return 1 2020-03-11T22:43:43.576 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:341: TEST_divergent_ec: echo 'delay recovery' 2020-03-11T22:43:43.576 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:342: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:43:43.576 INFO:tasks.workunit.client.0.smithi059.stdout:delay recovery 2020-03-11T22:43:43.751 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:43:43.752 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 101 0 101 0 0 48430 0 0 18 18 active+undersized+degraded 2020-03-11T22:43:07.788990+0000 39'102 45:135 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2020-03-11T22:41:33.943641+0000 0'0 2020-03-11T22:41:33.943641+0000 0 2020-03-11T22:43:43.752 INFO:tasks.workunit.client.0.smithi059.stdout: 2020-03-11T22:43:43.752 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:43:43.753 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2020-03-11T22:43:43.761 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:343: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:43:43.762 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: get_asok_path osd.0 2020-03-11T22:43:43.762 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-11T22:43:43.762 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-11T22:43:43.762 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:43:43.763 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:43:43.763 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:43:43.763 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17931/ceph-osd.0.asok 2020-03-11T22:43:43.763 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: CEPH_ARGS= 2020-03-11T22:43:43.764 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.17931/ceph-osd.0.asok set_recovery_delay 100000 2020-03-11T22:43:43.859 INFO:tasks.workunit.client.0.smithi059.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:43:43.866 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:343: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:43:43.867 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: get_asok_path osd.2 2020-03-11T22:43:43.867 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-03-11T22:43:43.867 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-03-11T22:43:43.868 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:43:43.868 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:43:43.868 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:43:43.868 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17931/ceph-osd.2.asok 2020-03-11T22:43:43.868 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: CEPH_ARGS= 2020-03-11T22:43:43.868 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.17931/ceph-osd.2.asok set_recovery_delay 100000 2020-03-11T22:43:43.964 INFO:tasks.workunit.client.0.smithi059.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:43:43.972 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:349: TEST_divergent_ec: echo 'revive divergent 1' 2020-03-11T22:43:43.972 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:350: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:43:43.972 INFO:tasks.workunit.client.0.smithi059.stdout:revive divergent 1 2020-03-11T22:43:44.312 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:43:44.312 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 101 0 101 0 0 48430 0 0 18 18 active+undersized+degraded 2020-03-11T22:43:07.788990+0000 39'102 45:135 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2020-03-11T22:41:33.943641+0000 0'0 2020-03-11T22:41:33.943641+0000 0 2020-03-11T22:43:44.312 INFO:tasks.workunit.client.0.smithi059.stdout: 2020-03-11T22:43:44.312 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:43:44.313 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2020-03-11T22:43:44.321 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:351: TEST_divergent_ec: ceph osd set noup 2020-03-11T22:43:44.946 INFO:tasks.workunit.client.0.smithi059.stderr:noup is set 2020-03-11T22:43:44.957 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:352: TEST_divergent_ec: activate_osd td/divergent-priors 1 2020-03-11T22:43:44.957 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:43:44.957 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:43:44.957 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-11T22:43:44.958 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:43:44.958 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:43:44.958 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=cb7f2e42-1934-4145-a3c3-29bdd93f8adb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:43:44.958 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:43:44.958 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:43:44.958 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:43:44.958 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:43:44.959 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:43:44.959 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:43:44.959 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:43:44.959 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:43:44.959 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:43:44.959 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:43:44.959 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:43:44.960 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:43:44.960 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:43:44.960 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:43:44.960 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17931/$cluster-$name.asok' 2020-03-11T22:43:44.960 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' 2020-03-11T22:43:44.960 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:43:44.960 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:43:44.961 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:43:44.961 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:43:44.961 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:43:44.961 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:43:44.961 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:43:44.961 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:43:44.961 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:43:44.962 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-11T22:43:44.962 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=cb7f2e42-1934-4145-a3c3-29bdd93f8adb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:43:44.962 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-03-11T22:43:44.962 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.1 2020-03-11T22:43:44.963 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-11T22:43:44.963 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:43:44.963 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:43:44.963 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:43:44.984 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:43:44.981+0000 7f1128e08ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:44.986 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:43:44.983+0000 7f1128e08ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:44.988 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:43:44.984+0000 7f1128e08ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:45.329 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: return 0 2020-03-11T22:43:45.329 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:353: TEST_divergent_ec: sleep 5 2020-03-11T22:43:45.504 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:43:45.501+0000 7f1128e08ec0 -1 Falling back to public interface 2020-03-11T22:43:46.372 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:43:46.368+0000 7f1128e08ec0 -1 osd.1 29 log_to_monitors {default=true} 2020-03-11T22:43:50.330 INFO:tasks.workunit.client.0.smithi059.stdout:delay recovery divergent 2020-03-11T22:43:50.330 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:355: TEST_divergent_ec: echo 'delay recovery divergent' 2020-03-11T22:43:50.330 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:356: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:43:50.686 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:43:50.686 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 101 0 101 0 0 48430 0 0 18 18 active+undersized+degraded 2020-03-11T22:43:07.788990+0000 39'102 45:135 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2020-03-11T22:41:33.943641+0000 0'0 2020-03-11T22:41:33.943641+0000 0 2020-03-11T22:43:50.687 INFO:tasks.workunit.client.0.smithi059.stdout: 2020-03-11T22:43:50.687 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:43:50.688 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2020-03-11T22:43:50.697 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: get_asok_path osd.1 2020-03-11T22:43:50.697 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:43:50.697 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:43:50.697 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:43:50.697 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:43:50.697 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:43:50.698 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17931/ceph-osd.1.asok 2020-03-11T22:43:50.698 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: CEPH_ARGS= 2020-03-11T22:43:50.698 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.17931/ceph-osd.1.asok set_recovery_delay 100000 2020-03-11T22:43:50.794 INFO:tasks.workunit.client.0.smithi059.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:43:50.801 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:359: TEST_divergent_ec: ceph osd unset noup 2020-03-11T22:43:51.577 INFO:tasks.workunit.client.0.smithi059.stderr:noup is unset 2020-03-11T22:43:51.585 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:361: TEST_divergent_ec: wait_for_osd up 0 2020-03-11T22:43:51.586 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:43:51.586 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:43:51.586 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:43:51.586 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:43:51.586 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:43:51.586 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:43:51.586 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:43:51.587 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:43:51.587 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:43:52.011 INFO:tasks.workunit.client.0.smithi059.stdout:osd.0 up in weight 1 up_from 41 up_thru 44 down_at 40 last_clean_interval [35,39) [v2:127.0.0.1:6802/50787,v1:127.0.0.1:6803/50787] [v2:127.0.0.1:6804/50787,v1:127.0.0.1:6805/50787] exists,up af7f49a6-1c6c-446f-802f-d448ee247966 2020-03-11T22:43:52.011 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:43:52.012 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:43:52.012 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:43:52.012 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:43:52.012 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:362: TEST_divergent_ec: wait_for_osd up 1 2020-03-11T22:43:52.012 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:43:52.012 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:43:52.013 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:43:52.013 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:43:52.013 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:43:52.013 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:43:52.013 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:43:52.013 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:43:52.375 INFO:tasks.workunit.client.0.smithi059.stdout:osd.1 up in weight 1 up_from 49 up_thru 49 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/52415,v1:127.0.0.1:6819/52415] [v2:127.0.0.1:6820/52415,v1:127.0.0.1:6821/52415] exists,up 01a46a03-2514-4118-9d87-d9feba7ccc07 2020-03-11T22:43:52.375 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:43:52.376 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:43:52.376 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:43:52.376 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:43:52.376 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:363: TEST_divergent_ec: wait_for_osd up 2 2020-03-11T22:43:52.376 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:43:52.376 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:43:52.377 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:43:52.377 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:43:52.377 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:43:52.377 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:43:52.377 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:43:52.377 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:43:52.754 INFO:tasks.workunit.client.0.smithi059.stdout:osd.2 up in weight 1 up_from 44 up_thru 0 down_at 43 last_clean_interval [36,39) [v2:127.0.0.1:6810/51019,v1:127.0.0.1:6811/51019] [v2:127.0.0.1:6812/51019,v1:127.0.0.1:6813/51019] exists,up d2482f9a-6ef9-4322-9e82-b5d8b8810499 2020-03-11T22:43:52.754 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:43:52.754 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:43:52.754 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:43:52.754 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:365: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:43:53.125 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:43:53.125 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 101 0 101 0 0 48430 0 0 18 18 active+undersized+degraded 2020-03-11T22:43:07.788990+0000 39'102 45:136 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2020-03-11T22:41:33.943641+0000 0'0 2020-03-11T22:41:33.943641+0000 0 2020-03-11T22:43:53.125 INFO:tasks.workunit.client.0.smithi059.stdout: 2020-03-11T22:43:53.125 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:43:53.126 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2020-03-11T22:43:53.136 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:366: TEST_divergent_ec: echo 'wait for peering' 2020-03-11T22:43:53.136 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:367: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:43:53.136 INFO:tasks.workunit.client.0.smithi059.stdout:wait for peering 2020-03-11T22:43:53.473 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:43:53.473 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 101 0 101 0 0 48430 0 0 18 18 active+undersized+degraded 2020-03-11T22:43:07.788990+0000 39'102 45:136 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2020-03-11T22:41:33.943641+0000 0'0 2020-03-11T22:41:33.943641+0000 0 2020-03-11T22:43:53.473 INFO:tasks.workunit.client.0.smithi059.stdout: 2020-03-11T22:43:53.473 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:43:53.475 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2020-03-11T22:43:53.483 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:368: TEST_divergent_ec: rados -p test put foo /etc/fstab 2020-03-11T22:44:03.838 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:370: TEST_divergent_ec: echo 'killing divergent 1' 2020-03-11T22:44:03.839 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:371: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:44:03.839 INFO:tasks.workunit.client.0.smithi059.stdout:killing divergent 1 2020-03-11T22:44:04.186 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:44:04.186 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 101 4 4 0 0 48430 0 0 12 12 active+recovering+degraded+wait 2020-03-11T22:43:51.916533+0000 39'102 51:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:33.943641+0000 0'0 2020-03-11T22:41:33.943641+0000 0 2020-03-11T22:44:04.186 INFO:tasks.workunit.client.0.smithi059.stdout: 2020-03-11T22:44:04.186 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:44:04.187 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2020-03-11T22:44:04.195 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:372: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.1 2020-03-11T22:44:04.196 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:44:04.196 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:44:04.196 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:44:04.197 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:44:04.197 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:44:04.301 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:44:04.301 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:374: TEST_divergent_ec: echo 'reviving divergent 1' 2020-03-11T22:44:04.302 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:375: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:44:04.302 INFO:tasks.workunit.client.0.smithi059.stdout:reviving divergent 1 2020-03-11T22:44:04.639 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:44:04.640 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 101 4 4 0 0 48430 0 0 12 12 active+recovering+degraded+wait 2020-03-11T22:43:51.916533+0000 39'102 51:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:33.943641+0000 0'0 2020-03-11T22:41:33.943641+0000 0 2020-03-11T22:44:04.640 INFO:tasks.workunit.client.0.smithi059.stdout: 2020-03-11T22:44:04.640 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:44:04.641 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2020-03-11T22:44:04.653 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:376: TEST_divergent_ec: activate_osd td/divergent-priors 1 2020-03-11T22:44:04.654 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:44:04.654 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:44:04.654 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-11T22:44:04.654 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:44:04.654 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:44:04.654 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=cb7f2e42-1934-4145-a3c3-29bdd93f8adb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:44:04.655 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:44:04.655 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:44:04.655 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:44:04.655 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:44:04.655 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:44:04.655 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:44:04.656 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:44:04.656 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:44:04.656 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:44:04.656 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:44:04.656 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:44:04.656 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:44:04.656 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:44:04.657 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:44:04.657 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17931/$cluster-$name.asok' 2020-03-11T22:44:04.657 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' 2020-03-11T22:44:04.657 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:44:04.657 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:44:04.658 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:44:04.658 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:44:04.658 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:44:04.658 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:44:04.658 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:44:04.659 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:44:04.659 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:44:04.659 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-11T22:44:04.659 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=cb7f2e42-1934-4145-a3c3-29bdd93f8adb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17931/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:44:04.659 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-03-11T22:44:04.659 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-11T22:44:04.660 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:44:04.660 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:44:04.660 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:44:04.660 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.1 2020-03-11T22:44:04.680 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:44:04.677+0000 7f1acd648ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:04.681 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:44:04.679+0000 7f1acd648ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:04.682 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:44:04.680+0000 7f1acd648ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:05.021 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 1 2020-03-11T22:44:05.021 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:44:05.022 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:44:05.022 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:44:05.022 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:44:05.022 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:44:05.022 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:44:05.022 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:44:05.023 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:44:05.023 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:44:05.201 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:44:05.198+0000 7f1acd648ec0 -1 Falling back to public interface 2020-03-11T22:44:05.401 INFO:tasks.workunit.client.0.smithi059.stdout:osd.1 up in weight 1 up_from 49 up_thru 49 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/52415,v1:127.0.0.1:6819/52415] [v2:127.0.0.1:6820/52415,v1:127.0.0.1:6821/52415] exists,up 01a46a03-2514-4118-9d87-d9feba7ccc07 2020-03-11T22:44:05.401 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:44:05.401 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:44:05.401 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:44:05.401 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:378: TEST_divergent_ec: sleep 20 2020-03-11T22:44:06.069 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:44:06.067+0000 7f1acd648ec0 -1 osd.1 51 log_to_monitors {default=true} 2020-03-11T22:44:25.402 INFO:tasks.workunit.client.0.smithi059.stdout:allowing recovery 2020-03-11T22:44:25.402 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:380: TEST_divergent_ec: echo 'allowing recovery' 2020-03-11T22:44:25.402 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:381: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:44:25.753 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:44:25.753 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 102 0 0 0 0 48913 0 0 13 13 active+clean 2020-03-11T22:44:07.207486+0000 51'103 54:153 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:33.943641+0000 0'0 2020-03-11T22:41:33.943641+0000 0 2020-03-11T22:44:25.754 INFO:tasks.workunit.client.0.smithi059.stdout: 2020-03-11T22:44:25.754 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:44:25.755 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2020-03-11T22:44:25.763 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2020-03-11T22:44:25.763 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.0 debug kick_recovery_wq 0 2020-03-11T22:44:25.891 INFO:tasks.workunit.client.0.smithi059.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:44:25.900 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2020-03-11T22:44:25.900 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.1 debug kick_recovery_wq 0 2020-03-11T22:44:26.026 INFO:tasks.workunit.client.0.smithi059.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:44:26.034 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2020-03-11T22:44:26.034 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.2 debug kick_recovery_wq 0 2020-03-11T22:44:26.161 INFO:tasks.workunit.client.0.smithi059.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:44:26.169 INFO:tasks.workunit.client.0.smithi059.stdout:reading divergent objects 2020-03-11T22:44:26.169 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:388: TEST_divergent_ec: echo 'reading divergent objects' 2020-03-11T22:44:26.169 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:389: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:44:26.507 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:44:26.507 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 102 0 0 0 0 48913 0 0 13 13 active+clean 2020-03-11T22:44:07.207486+0000 51'103 54:153 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:41:33.943641+0000 0'0 2020-03-11T22:41:33.943641+0000 0 2020-03-11T22:44:26.507 INFO:tasks.workunit.client.0.smithi059.stdout: 2020-03-11T22:44:26.507 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:44:26.508 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2020-03-11T22:44:26.517 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: expr 2 + 2 2020-03-11T22:44:26.518 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: seq 1 4 2020-03-11T22:44:26.519 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:44:26.519 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_1 td/divergent-priors/existing 2020-03-11T22:44:26.572 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:44:26.572 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_2 td/divergent-priors/existing 2020-03-11T22:44:26.602 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:44:26.602 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_3 td/divergent-priors/existing 2020-03-11T22:44:26.632 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:44:26.632 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_4 td/divergent-priors/existing 2020-03-11T22:44:26.661 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:394: TEST_divergent_ec: rm -f td/divergent-priors/existing 2020-03-11T22:44:26.662 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:396: TEST_divergent_ec: find td/divergent-priors -name '*osd*log' 2020-03-11T22:44:26.663 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:396: TEST_divergent_ec: grep _merge_object_divergent_entries td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:44:26.669 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:43:51.863+0000 7f1106fc3700 20 _merge_object_divergent_entries: merging hoid 1:7a80c0d5:::existing_99:1 entries: 29'103 (28'99) clone 1:7a80c0d5:::existing_99:1 by unknown.0.0:0 2020-03-11T22:41:54.743220+0000 0 snaps [1] ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:44:26.669 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:43:51.863+0000 7f1106fc3700 20 _merge_object_divergent_entries: keeping 29'103 (28'99) clone 1:7a80c0d5:::existing_99:1 by unknown.0.0:0 2020-03-11T22:41:54.743220+0000 0 snaps [1] ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:44:26.669 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:43:51.863+0000 7f1106fc3700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:44:26.669 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:43:51.863+0000 7f1106fc3700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 prior_version: 28'99 first_divergent_update: 29'103 last_divergent_update: 29'103 2020-03-11T22:44:26.669 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:43:51.863+0000 7f1106fc3700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 has no more recent entries in log 2020-03-11T22:44:26.669 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:43:51.863+0000 7f1106fc3700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 prior_version or op type indicates creation, deleting 2020-03-11T22:44:26.670 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:43:51.863+0000 7f1106fc3700 20 _merge_object_divergent_entries: merging hoid 1:7a80c0d5:::existing_99:head entries: 29'104 (28'99) modify 1:7a80c0d5:::existing_99:head by client.4555.0:1 2020-03-11T22:42:01.636064+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:44:26.670 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:43:51.863+0000 7f1106fc3700 20 _merge_object_divergent_entries: keeping 29'104 (28'99) modify 1:7a80c0d5:::existing_99:head by client.4555.0:1 2020-03-11T22:42:01.636064+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:44:26.670 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:43:51.863+0000 7f1106fc3700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:44:26.670 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:43:51.863+0000 7f1106fc3700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head prior_version: 28'99 first_divergent_update: 29'104 last_divergent_update: 29'104 2020-03-11T22:44:26.670 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:43:51.863+0000 7f1106fc3700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head has no more recent entries in log 2020-03-11T22:44:26.671 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:43:51.863+0000 7f1106fc3700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head must be rolled back or recovered, attempting to rollback 2020-03-11T22:44:26.671 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:43:51.863+0000 7f1106fc3700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head olog_can_rollback_to: 29'104 2020-03-11T22:44:26.671 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:43:51.863+0000 7f1106fc3700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head cannot rollback 29'104 (28'99) modify 1:7a80c0d5:::existing_99:head by client.4555.0:1 2020-03-11T22:42:01.636064+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:44:26.671 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:43:51.863+0000 7f1106fc3700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head cannot roll back, removing and adding to missing 2020-03-11T22:44:26.671 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:43:51.863+0000 7f1106fc3700 20 _merge_object_divergent_entries: merging hoid 1:c156033e:::existing_100:head entries: 28'101 (28'100) modify 1:c156033e:::existing_100:head by client.4546.0:1 2020-03-11T22:42:00.465461+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0,28'102 (28'101) modify 1:c156033e:::existing_100:head by client.4552.0:1 2020-03-11T22:42:01.471848+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:44:26.671 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:43:51.863+0000 7f1106fc3700 20 _merge_object_divergent_entries: keeping 28'101 (28'100) modify 1:c156033e:::existing_100:head by client.4546.0:1 2020-03-11T22:42:00.465461+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:44:26.671 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:43:51.863+0000 7f1106fc3700 20 _merge_object_divergent_entries: keeping 28'102 (28'101) modify 1:c156033e:::existing_100:head by client.4552.0:1 2020-03-11T22:42:01.471848+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:44:26.672 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:43:51.863+0000 7f1106fc3700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:44:26.672 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:43:51.863+0000 7f1106fc3700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head prior_version: 28'100 first_divergent_update: 28'101 last_divergent_update: 28'102 2020-03-11T22:44:26.672 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:43:51.863+0000 7f1106fc3700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head has no more recent entries in log 2020-03-11T22:44:26.672 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:43:51.863+0000 7f1106fc3700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head must be rolled back or recovered, attempting to rollback 2020-03-11T22:44:26.672 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:43:51.863+0000 7f1106fc3700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head olog_can_rollback_to: 29'104 2020-03-11T22:44:26.672 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:43:51.863+0000 7f1106fc3700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot rollback 28'102 (28'101) modify 1:c156033e:::existing_100:head by client.4552.0:1 2020-03-11T22:42:01.471848+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:44:26.672 INFO:tasks.workunit.client.0.smithi059.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:43:51.863+0000 7f1106fc3700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot roll back, removing and adding to missing 2020-03-11T22:44:26.673 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:405: TEST_divergent_ec: find td/divergent-priors -name '*osd*log' 2020-03-11T22:44:26.674 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:405: TEST_divergent_ec: grep -q '_merge_object_divergent_entries.*rolled back' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:44:26.674 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:410: TEST_divergent_ec: echo success 2020-03-11T22:44:26.674 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:412: TEST_divergent_ec: delete_pool test 2020-03-11T22:44:26.674 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test 2020-03-11T22:44:26.674 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2020-03-11T22:44:26.674 INFO:tasks.workunit.client.0.smithi059.stdout:success 2020-03-11T22:44:27.220 INFO:tasks.workunit.client.0.smithi059.stderr:pool 'test' does not exist 2020-03-11T22:44:27.227 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:413: TEST_divergent_ec: kill_daemons td/divergent-priors 2020-03-11T22:44:27.228 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:44:27.228 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:44:27.228 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:44:27.229 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:44:27.229 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:44:27.336 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:44:27.337 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2020-03-11T22:44:27.337 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-11T22:44:27.337 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:44:27.337 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-11T22:44:27.337 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:44:27.337 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:44:27.338 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:44:27.338 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:44:27.338 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:44:27.342 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:44:27.342 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:44:27.343 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:44:27.343 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:44:27.344 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:44:27.344 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:44:27.344 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:44:27.346 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:44:27.346 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:44:27.346 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:44:27.347 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:44:27.347 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:44:27.348 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:44:27.348 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-11T22:44:27.361 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:44:27.361 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:44:27.361 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:44:27.361 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17931 2020-03-11T22:44:27.363 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:44:27.363 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:44:27.363 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: code=0 2020-03-11T22:44:27.363 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: teardown td/divergent-priors 0 2020-03-11T22:44:27.364 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-11T22:44:27.364 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2020-03-11T22:44:27.364 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-11T22:44:27.364 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:44:27.364 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:44:27.364 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:44:27.364 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:44:27.365 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:44:27.365 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:44:27.365 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:44:27.366 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:44:27.366 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:44:27.366 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:44:27.367 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:44:27.367 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:44:27.369 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:44:27.369 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:44:27.369 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:44:27.370 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:44:27.370 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:44:27.371 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2020-03-11T22:44:27.371 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-11T22:44:27.372 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:44:27.372 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:44:27.372 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17931 2020-03-11T22:44:27.372 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17931 2020-03-11T22:44:27.374 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:44:27.374 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:44:27.374 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: return 0 2020-03-11T22:44:27.374 INFO:teuthology.orchestra.run:Running command with timeout 3600 2020-03-11T22:44:27.375 INFO:teuthology.orchestra.run.smithi059:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2020-03-11T22:44:27.443 INFO:tasks.workunit:Running workunit osd/ec-error-rollforward.sh... 2020-03-11T22:44:27.443 INFO:teuthology.orchestra.run.smithi059:workunit test osd/ec-error-rollforward.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4acc1121e47039a3cdf2978ae458c10d153aa141 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh 2020-03-11T22:44:27.476 INFO:tasks.workunit.client.0.smithi059.stderr:stty: 'standard input': Inappropriate ioctl for device 2020-03-11T22:44:27.484 INFO:tasks.workunit.client.0.smithi059.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2020-03-11T22:44:27.484 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2020-03-11T22:44:27.484 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2020-03-11T22:44:27.484 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: main: export PYTHONWARNINGS=ignore 2020-03-11T22:44:27.484 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: main: PYTHONWARNINGS=ignore 2020-03-11T22:44:27.485 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: main: export CEPH_CONF=/dev/null 2020-03-11T22:44:27.485 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: main: CEPH_CONF=/dev/null 2020-03-11T22:44:27.485 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: main: unset CEPH_ARGS 2020-03-11T22:44:27.485 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: local code 2020-03-11T22:44:27.485 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: run td/ec-error-rollforward 2020-03-11T22:44:27.485 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:6: run: local dir=td/ec-error-rollforward 2020-03-11T22:44:27.485 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:7: run: shift 2020-03-11T22:44:27.486 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:10: run: export CEPH_MON=127.0.0.1:7132 2020-03-11T22:44:27.486 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:10: run: CEPH_MON=127.0.0.1:7132 2020-03-11T22:44:27.486 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:11: run: export CEPH_ARGS 2020-03-11T22:44:27.486 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:12: run: uuidgen 2020-03-11T22:44:27.486 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:12: run: CEPH_ARGS+='--fsid=c7b6a6f8-a09d-4af9-b123-d357c2226ef8 --auth-supported=none ' 2020-03-11T22:44:27.486 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:13: run: CEPH_ARGS+='--mon-host=127.0.0.1:7132 ' 2020-03-11T22:44:27.486 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:14: run: export margin=10 2020-03-11T22:44:27.487 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:14: run: margin=10 2020-03-11T22:44:27.487 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:15: run: export objects=200 2020-03-11T22:44:27.487 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:15: run: objects=200 2020-03-11T22:44:27.487 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:16: run: export poolname=test 2020-03-11T22:44:27.487 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:16: run: poolname=test 2020-03-11T22:44:27.487 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: set 2020-03-11T22:44:27.487 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2020-03-11T22:44:27.489 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: local funcs=TEST_ec_error_rollforward 2020-03-11T22:44:27.490 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:19: run: for func in $funcs 2020-03-11T22:44:27.490 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:20: run: setup td/ec-error-rollforward 2020-03-11T22:44:27.490 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ec-error-rollforward 2020-03-11T22:44:27.490 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ec-error-rollforward 2020-03-11T22:44:27.490 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2020-03-11T22:44:27.490 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:44:27.490 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2020-03-11T22:44:27.491 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:44:27.491 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:44:27.491 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:44:27.491 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:44:27.491 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:44:27.491 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:44:27.492 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:44:27.492 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:44:27.492 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:44:27.492 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:44:27.492 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:44:27.492 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:44:27.494 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:44:27.494 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:44:27.494 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:44:27.494 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:44:27.494 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:44:27.494 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:44:27.494 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2020-03-11T22:44:27.495 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:44:27.495 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:44:27.495 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53335 2020-03-11T22:44:27.495 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.53335 2020-03-11T22:44:27.496 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:44:27.496 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:44:27.496 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ec-error-rollforward 2020-03-11T22:44:27.496 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-11T22:44:27.497 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:44:27.497 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53335 2020-03-11T22:44:27.497 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.53335 2020-03-11T22:44:27.498 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-11T22:44:27.499 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2020-03-11T22:44:27.499 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2020-03-11T22:44:27.499 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-11T22:44:27.499 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ec-error-rollforward 1' TERM HUP INT 2020-03-11T22:44:27.499 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:21: run: TEST_ec_error_rollforward td/ec-error-rollforward 2020-03-11T22:44:27.499 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:27: TEST_ec_error_rollforward: local dir=td/ec-error-rollforward 2020-03-11T22:44:27.499 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:28: TEST_ec_error_rollforward: run_mon td/ec-error-rollforward a 2020-03-11T22:44:27.500 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ec-error-rollforward 2020-03-11T22:44:27.500 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-11T22:44:27.500 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-11T22:44:27.500 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-11T22:44:27.500 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ec-error-rollforward/a 2020-03-11T22:44:27.500 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ec-error-rollforward/a --run-dir=td/ec-error-rollforward 2020-03-11T22:44:27.774 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-11T22:44:27.775 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:44:27.775 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:44:27.775 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:44:27.776 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:44:27.776 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53335 2020-03-11T22:44:27.776 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53335/$cluster-$name.asok' 2020-03-11T22:44:27.776 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ec-error-rollforward/a '--log-file=td/ec-error-rollforward/$name.log' '--admin-socket=/tmp/ceph-asok.53335/$cluster-$name.asok' --mon-cluster-log-file=td/ec-error-rollforward/log --run-dir=td/ec-error-rollforward '--pid-file=td/ec-error-rollforward/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-11T22:44:27.926 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-11T22:44:27.927 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-11T22:44:27.928 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:44:27.928 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:44:27.928 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-11T22:44:27.928 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-11T22:44:27.928 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:44:27.928 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:44:27.928 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:44:27.929 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:44:27.929 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:44:27.929 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53335 2020-03-11T22:44:27.929 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53335/ceph-mon.a.asok 2020-03-11T22:44:27.929 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:44:27.929 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.53335/ceph-mon.a.asok config get fsid 2020-03-11T22:44:28.019 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-11T22:44:28.019 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:44:28.020 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:44:28.020 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-11T22:44:28.020 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-11T22:44:28.020 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:44:28.020 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:44:28.020 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:44:28.020 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:44:28.021 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:44:28.021 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53335 2020-03-11T22:44:28.021 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53335/ceph-mon.a.asok 2020-03-11T22:44:28.021 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:44:28.021 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.53335/ceph-mon.a.asok config get mon_host 2020-03-11T22:44:28.110 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:29: TEST_ec_error_rollforward: run_mgr td/ec-error-rollforward x 2020-03-11T22:44:28.110 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/ec-error-rollforward 2020-03-11T22:44:28.110 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-11T22:44:28.110 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-11T22:44:28.111 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-11T22:44:28.111 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/ec-error-rollforward/x 2020-03-11T22:44:28.111 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-11T22:44:28.995 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-11T22:44:28.995 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:44:28.995 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:44:28.995 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:44:28.995 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:44:28.995 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53335 2020-03-11T22:44:28.995 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53335/$cluster-$name.asok' 2020-03-11T22:44:28.996 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:44:28.996 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ec-error-rollforward/x '--log-file=td/ec-error-rollforward/$name.log' '--admin-socket=/tmp/ceph-asok.53335/$cluster-$name.asok' --run-dir=td/ec-error-rollforward '--pid-file=td/ec-error-rollforward/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:44:29.018 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:30: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 0 2020-03-11T22:44:29.018 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/ec-error-rollforward 2020-03-11T22:44:29.019 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:44:29.019 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-11T22:44:29.019 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:44:29.019 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/ec-error-rollforward/0 2020-03-11T22:44:29.019 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=c7b6a6f8-a09d-4af9-b123-d357c2226ef8 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-03-11T22:44:29.019 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:44:29.020 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:44:29.020 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:44:29.020 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/0' 2020-03-11T22:44:29.020 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/0/journal' 2020-03-11T22:44:29.020 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:44:29.020 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:44:29.020 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-03-11T22:44:29.020 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:44:29.021 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:44:29.021 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:44:29.021 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:44:29.021 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:44:29.021 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53335 2020-03-11T22:44:29.021 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53335/$cluster-$name.asok' 2020-03-11T22:44:29.022 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53335/$cluster-$name.asok' 2020-03-11T22:44:29.022 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:44:29.022 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:44:29.022 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:44:29.022 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-03-11T22:44:29.022 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-03-11T22:44:29.023 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:44:29.023 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:44:29.023 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:44:29.023 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:44:29.023 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:44:29.023 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/ec-error-rollforward/0 2020-03-11T22:44:29.024 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:44:29.024 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=8355d9fe-0420-40f1-be0d-c6a59a7a9ced 2020-03-11T22:44:29.025 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 8355d9fe-0420-40f1-be0d-c6a59a7a9ced' 2020-03-11T22:44:29.025 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:44:29.025 INFO:tasks.workunit.client.0.smithi059.stdout:add osd0 8355d9fe-0420-40f1-be0d-c6a59a7a9ced 2020-03-11T22:44:29.032 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQDNaWlet3C+ARAAfPV5xClw9ocbLWgpIOnU8w== 2020-03-11T22:44:29.032 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQDNaWlet3C+ARAAfPV5xClw9ocbLWgpIOnU8w=="}' 2020-03-11T22:44:29.032 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 8355d9fe-0420-40f1-be0d-c6a59a7a9ced -i td/ec-error-rollforward/0/new.json 2020-03-11T22:44:29.384 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:44:29.394 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/ec-error-rollforward/0/new.json 2020-03-11T22:44:29.394 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=c7b6a6f8-a09d-4af9-b123-d357c2226ef8 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.53335/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDNaWlet3C+ARAAfPV5xClw9ocbLWgpIOnU8w== --osd-uuid 8355d9fe-0420-40f1-be0d-c6a59a7a9ced 2020-03-11T22:44:29.415 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:44:29.413+0000 7f7780a2aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:29.417 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:44:29.414+0000 7f7780a2aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:29.418 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:44:29.416+0000 7f7780a2aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:29.419 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:44:29.416+0000 7f7780a2aec0 -1 bluestore(td/ec-error-rollforward/0/block) _read_bdev_label failed to open td/ec-error-rollforward/0/block: (2) No such file or directory 2020-03-11T22:44:29.419 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:44:29.416+0000 7f7780a2aec0 -1 bluestore(td/ec-error-rollforward/0/block) _read_bdev_label failed to open td/ec-error-rollforward/0/block: (2) No such file or directory 2020-03-11T22:44:29.419 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:44:29.416+0000 7f7780a2aec0 -1 bluestore(td/ec-error-rollforward/0/block) _read_bdev_label failed to open td/ec-error-rollforward/0/block: (2) No such file or directory 2020-03-11T22:44:29.474 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:44:29.472+0000 7f7780a2aec0 -1 bluestore(td/ec-error-rollforward/0) _read_fsid unparsable uuid 2020-03-11T22:44:32.543 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/ec-error-rollforward/0/keyring 2020-03-11T22:44:32.544 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:44:32.544 INFO:tasks.workunit.client.0.smithi059.stdout:adding osd0 key to auth repository 2020-03-11T22:44:32.545 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-11T22:44:32.545 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/ec-error-rollforward/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:44:33.029 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-11T22:44:33.029 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=c7b6a6f8-a09d-4af9-b123-d357c2226ef8 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.53335/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:44:33.029 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:44:33.029 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:44:33.030 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:44:33.030 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.0 2020-03-11T22:44:33.047 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:44:33.045+0000 7f48dd14dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:33.049 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:44:33.047+0000 7f48dd14dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:33.051 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:44:33.048+0000 7f48dd14dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:33.446 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-11T22:44:33.446 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:44:33.446 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:44:33.446 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:44:33.446 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:44:33.446 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:44:33.446 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:44:33.447 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:44:33.447 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:44:33.447 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:44:33.569 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:44:33.564+0000 7f48dd14dec0 -1 Falling back to public interface 2020-03-11T22:44:33.808 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:44:34.340 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:44:34.338+0000 7f48dd14dec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-11T22:44:34.809 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:44:34.810 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:44:34.810 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:44:34.810 INFO:tasks.workunit.client.0.smithi059.stdout:1 2020-03-11T22:44:34.810 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:44:34.811 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:44:35.212 INFO:tasks.workunit.client.0.smithi059.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/53764,v1:127.0.0.1:6803/53764] [v2:127.0.0.1:6804/53764,v1:127.0.0.1:6805/53764] exists,up 8355d9fe-0420-40f1-be0d-c6a59a7a9ced 2020-03-11T22:44:35.212 INFO:tasks.workunit.client.0.smithi059.stdout:add osd1 a9e70d5d-6bd3-4537-8695-a71a2a70af22 2020-03-11T22:44:35.212 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:44:35.212 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:44:35.212 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:44:35.213 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:31: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 1 2020-03-11T22:44:35.213 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/ec-error-rollforward 2020-03-11T22:44:35.213 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:44:35.213 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-11T22:44:35.213 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:44:35.213 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/ec-error-rollforward/1 2020-03-11T22:44:35.213 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=c7b6a6f8-a09d-4af9-b123-d357c2226ef8 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-03-11T22:44:35.214 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:44:35.214 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:44:35.214 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:44:35.214 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/1' 2020-03-11T22:44:35.214 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/1/journal' 2020-03-11T22:44:35.214 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:44:35.214 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:44:35.214 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-03-11T22:44:35.215 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:44:35.215 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:44:35.215 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:44:35.215 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:44:35.215 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:44:35.215 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53335 2020-03-11T22:44:35.215 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53335/$cluster-$name.asok' 2020-03-11T22:44:35.216 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53335/$cluster-$name.asok' 2020-03-11T22:44:35.216 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:44:35.216 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:44:35.216 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:44:35.216 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-03-11T22:44:35.217 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-03-11T22:44:35.217 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:44:35.217 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:44:35.217 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:44:35.217 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:44:35.217 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:44:35.217 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/ec-error-rollforward/1 2020-03-11T22:44:35.218 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:44:35.218 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=a9e70d5d-6bd3-4537-8695-a71a2a70af22 2020-03-11T22:44:35.218 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 a9e70d5d-6bd3-4537-8695-a71a2a70af22' 2020-03-11T22:44:35.218 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:44:35.225 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQDTaWlesaBCDRAAFTSZaYAE56ATceqCIl5E3Q== 2020-03-11T22:44:35.225 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQDTaWlesaBCDRAAFTSZaYAE56ATceqCIl5E3Q=="}' 2020-03-11T22:44:35.225 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new a9e70d5d-6bd3-4537-8695-a71a2a70af22 -i td/ec-error-rollforward/1/new.json 2020-03-11T22:44:35.702 INFO:tasks.workunit.client.0.smithi059.stdout:1 2020-03-11T22:44:35.711 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/ec-error-rollforward/1/new.json 2020-03-11T22:44:35.712 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=c7b6a6f8-a09d-4af9-b123-d357c2226ef8 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.53335/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDTaWlesaBCDRAAFTSZaYAE56ATceqCIl5E3Q== --osd-uuid a9e70d5d-6bd3-4537-8695-a71a2a70af22 2020-03-11T22:44:35.733 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:44:35.730+0000 7fabdf263ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:35.734 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:44:35.732+0000 7fabdf263ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:35.737 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:44:35.734+0000 7fabdf263ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:35.737 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:44:35.734+0000 7fabdf263ec0 -1 bluestore(td/ec-error-rollforward/1/block) _read_bdev_label failed to open td/ec-error-rollforward/1/block: (2) No such file or directory 2020-03-11T22:44:35.737 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:44:35.734+0000 7fabdf263ec0 -1 bluestore(td/ec-error-rollforward/1/block) _read_bdev_label failed to open td/ec-error-rollforward/1/block: (2) No such file or directory 2020-03-11T22:44:35.737 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:44:35.734+0000 7fabdf263ec0 -1 bluestore(td/ec-error-rollforward/1/block) _read_bdev_label failed to open td/ec-error-rollforward/1/block: (2) No such file or directory 2020-03-11T22:44:35.788 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:44:35.786+0000 7fabdf263ec0 -1 bluestore(td/ec-error-rollforward/1) _read_fsid unparsable uuid 2020-03-11T22:44:39.142 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/ec-error-rollforward/1/keyring 2020-03-11T22:44:39.143 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:44:39.144 INFO:tasks.workunit.client.0.smithi059.stdout:adding osd1 key to auth repository 2020-03-11T22:44:39.144 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-03-11T22:44:39.144 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/ec-error-rollforward/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:44:39.628 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.1 2020-03-11T22:44:39.628 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-03-11T22:44:39.628 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:44:39.629 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:44:39.629 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:44:39.629 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=c7b6a6f8-a09d-4af9-b123-d357c2226ef8 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.53335/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:44:39.647 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:44:39.644+0000 7f280bfe3ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:39.649 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:44:39.646+0000 7f280bfe3ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:39.650 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:44:39.648+0000 7f280bfe3ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:39.989 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-03-11T22:44:39.990 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:44:39.990 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:44:39.990 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:44:39.990 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:44:39.990 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:44:39.990 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:44:39.990 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:44:39.991 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:44:39.991 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:44:40.168 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:44:40.163+0000 7f280bfe3ec0 -1 Falling back to public interface 2020-03-11T22:44:40.353 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:44:40.930 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:44:40.927+0000 7f280bfe3ec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-11T22:44:41.355 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:44:41.355 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:44:41.355 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:44:41.355 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:44:41.356 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:44:41.356 INFO:tasks.workunit.client.0.smithi059.stdout:1 2020-03-11T22:44:41.743 INFO:tasks.workunit.client.0.smithi059.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/54127,v1:127.0.0.1:6811/54127] [v2:127.0.0.1:6812/54127,v1:127.0.0.1:6813/54127] exists,up a9e70d5d-6bd3-4537-8695-a71a2a70af22 2020-03-11T22:44:41.743 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:44:41.743 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:44:41.744 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:44:41.744 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:32: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 2 2020-03-11T22:44:41.744 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/ec-error-rollforward 2020-03-11T22:44:41.744 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:44:41.744 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-03-11T22:44:41.744 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:44:41.744 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/ec-error-rollforward/2 2020-03-11T22:44:41.745 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=c7b6a6f8-a09d-4af9-b123-d357c2226ef8 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-03-11T22:44:41.745 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:44:41.745 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:44:41.745 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:44:41.745 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/2' 2020-03-11T22:44:41.745 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/2/journal' 2020-03-11T22:44:41.745 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:44:41.746 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:44:41.746 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-03-11T22:44:41.746 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:44:41.746 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:44:41.746 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:44:41.746 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:44:41.746 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:44:41.748 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53335 2020-03-11T22:44:41.749 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53335/$cluster-$name.asok' 2020-03-11T22:44:41.749 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53335/$cluster-$name.asok' 2020-03-11T22:44:41.749 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:44:41.749 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:44:41.749 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:44:41.749 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-03-11T22:44:41.749 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-03-11T22:44:41.750 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:44:41.750 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:44:41.750 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:44:41.750 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:44:41.750 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:44:41.750 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/ec-error-rollforward/2 2020-03-11T22:44:41.751 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:44:41.751 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=0e421bdb-2cd6-4049-b2dc-2be5fd0855e0 2020-03-11T22:44:41.751 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 0e421bdb-2cd6-4049-b2dc-2be5fd0855e0' 2020-03-11T22:44:41.751 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:44:41.752 INFO:tasks.workunit.client.0.smithi059.stdout:add osd2 0e421bdb-2cd6-4049-b2dc-2be5fd0855e0 2020-03-11T22:44:41.757 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQDZaWleAy4DLRAA5X0XcSP80UBiQchOm3bE0Q== 2020-03-11T22:44:41.758 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQDZaWleAy4DLRAA5X0XcSP80UBiQchOm3bE0Q=="}' 2020-03-11T22:44:41.758 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 0e421bdb-2cd6-4049-b2dc-2be5fd0855e0 -i td/ec-error-rollforward/2/new.json 2020-03-11T22:44:42.275 INFO:tasks.workunit.client.0.smithi059.stdout:2 2020-03-11T22:44:42.285 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/ec-error-rollforward/2/new.json 2020-03-11T22:44:42.286 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=c7b6a6f8-a09d-4af9-b123-d357c2226ef8 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.53335/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDZaWleAy4DLRAA5X0XcSP80UBiQchOm3bE0Q== --osd-uuid 0e421bdb-2cd6-4049-b2dc-2be5fd0855e0 2020-03-11T22:44:42.307 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:44:42.303+0000 7f6872e1bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:42.308 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:44:42.305+0000 7f6872e1bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:42.310 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:44:42.306+0000 7f6872e1bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:42.310 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:44:42.307+0000 7f6872e1bec0 -1 bluestore(td/ec-error-rollforward/2/block) _read_bdev_label failed to open td/ec-error-rollforward/2/block: (2) No such file or directory 2020-03-11T22:44:42.311 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:44:42.307+0000 7f6872e1bec0 -1 bluestore(td/ec-error-rollforward/2/block) _read_bdev_label failed to open td/ec-error-rollforward/2/block: (2) No such file or directory 2020-03-11T22:44:42.311 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:44:42.307+0000 7f6872e1bec0 -1 bluestore(td/ec-error-rollforward/2/block) _read_bdev_label failed to open td/ec-error-rollforward/2/block: (2) No such file or directory 2020-03-11T22:44:42.400 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:44:42.396+0000 7f6872e1bec0 -1 bluestore(td/ec-error-rollforward/2) _read_fsid unparsable uuid 2020-03-11T22:44:45.657 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/ec-error-rollforward/2/keyring 2020-03-11T22:44:45.658 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:44:45.659 INFO:tasks.workunit.client.0.smithi059.stdout:adding osd2 key to auth repository 2020-03-11T22:44:45.659 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-03-11T22:44:45.659 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/ec-error-rollforward/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:44:46.142 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.2 2020-03-11T22:44:46.143 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-03-11T22:44:46.143 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=c7b6a6f8-a09d-4af9-b123-d357c2226ef8 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.53335/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:44:46.143 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:44:46.143 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:44:46.143 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:44:46.161 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:44:46.157+0000 7fcee265cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:46.163 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:44:46.159+0000 7fcee265cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:46.165 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:44:46.161+0000 7fcee265cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:46.532 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-03-11T22:44:46.532 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:44:46.532 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:44:46.532 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:44:46.532 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:44:46.532 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:44:46.533 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:44:46.533 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:44:46.533 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:44:46.533 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:44:46.683 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:44:46.679+0000 7fcee265cec0 -1 Falling back to public interface 2020-03-11T22:44:46.914 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:44:47.427 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:44:47.424+0000 7fcee265cec0 -1 osd.2 0 log_to_monitors {default=true} 2020-03-11T22:44:47.915 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:44:47.916 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:44:47.916 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:44:47.916 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:44:47.916 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:44:47.916 INFO:tasks.workunit.client.0.smithi059.stdout:1 2020-03-11T22:44:48.300 INFO:tasks.workunit.client.0.smithi059.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/54482,v1:127.0.0.1:6819/54482] [v2:127.0.0.1:6820/54482,v1:127.0.0.1:6821/54482] exists,up 0e421bdb-2cd6-4049-b2dc-2be5fd0855e0 2020-03-11T22:44:48.301 INFO:tasks.workunit.client.0.smithi059.stdout:add osd3 b497abf9-591e-43e1-b78a-4ad9d9f2c595 2020-03-11T22:44:48.301 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:44:48.301 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:44:48.302 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:44:48.302 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:33: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 3 2020-03-11T22:44:48.302 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/ec-error-rollforward 2020-03-11T22:44:48.302 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:44:48.302 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=3 2020-03-11T22:44:48.303 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:44:48.303 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/ec-error-rollforward/3 2020-03-11T22:44:48.303 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=c7b6a6f8-a09d-4af9-b123-d357c2226ef8 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-03-11T22:44:48.303 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:44:48.304 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:44:48.304 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:44:48.304 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/3' 2020-03-11T22:44:48.304 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/3/journal' 2020-03-11T22:44:48.305 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:44:48.305 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:44:48.305 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-03-11T22:44:48.305 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:44:48.306 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:44:48.306 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:44:48.306 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:44:48.306 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:44:48.307 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53335 2020-03-11T22:44:48.307 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53335/$cluster-$name.asok' 2020-03-11T22:44:48.307 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53335/$cluster-$name.asok' 2020-03-11T22:44:48.307 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:44:48.308 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:44:48.308 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:44:48.308 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-03-11T22:44:48.308 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-03-11T22:44:48.309 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:44:48.309 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:44:48.309 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:44:48.309 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:44:48.310 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:44:48.310 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/ec-error-rollforward/3 2020-03-11T22:44:48.310 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:44:48.310 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=b497abf9-591e-43e1-b78a-4ad9d9f2c595 2020-03-11T22:44:48.311 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd3 b497abf9-591e-43e1-b78a-4ad9d9f2c595' 2020-03-11T22:44:48.311 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:44:48.312 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQDgaWleIBhxEhAAB0uCu3s3Weh6Vq/qzPf0og== 2020-03-11T22:44:48.312 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQDgaWleIBhxEhAAB0uCu3s3Weh6Vq/qzPf0og=="}' 2020-03-11T22:44:48.312 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new b497abf9-591e-43e1-b78a-4ad9d9f2c595 -i td/ec-error-rollforward/3/new.json 2020-03-11T22:44:48.880 INFO:tasks.workunit.client.0.smithi059.stdout:3 2020-03-11T22:44:48.890 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/ec-error-rollforward/3/new.json 2020-03-11T22:44:48.891 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 3 --fsid=c7b6a6f8-a09d-4af9-b123-d357c2226ef8 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.53335/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDgaWleIBhxEhAAB0uCu3s3Weh6Vq/qzPf0og== --osd-uuid b497abf9-591e-43e1-b78a-4ad9d9f2c595 2020-03-11T22:44:48.912 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:44:48.908+0000 7f0b1bbeaec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:48.960 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:44:48.957+0000 7f0b1bbeaec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:48.962 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:44:48.958+0000 7f0b1bbeaec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:48.963 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:44:48.959+0000 7f0b1bbeaec0 -1 bluestore(td/ec-error-rollforward/3/block) _read_bdev_label failed to open td/ec-error-rollforward/3/block: (2) No such file or directory 2020-03-11T22:44:48.963 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:44:48.959+0000 7f0b1bbeaec0 -1 bluestore(td/ec-error-rollforward/3/block) _read_bdev_label failed to open td/ec-error-rollforward/3/block: (2) No such file or directory 2020-03-11T22:44:48.963 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:44:48.959+0000 7f0b1bbeaec0 -1 bluestore(td/ec-error-rollforward/3/block) _read_bdev_label failed to open td/ec-error-rollforward/3/block: (2) No such file or directory 2020-03-11T22:44:49.018 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:44:49.015+0000 7f0b1bbeaec0 -1 bluestore(td/ec-error-rollforward/3) _read_fsid unparsable uuid 2020-03-11T22:44:52.102 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/ec-error-rollforward/3/keyring 2020-03-11T22:44:52.103 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:44:52.104 INFO:tasks.workunit.client.0.smithi059.stdout:adding osd3 key to auth repository 2020-03-11T22:44:52.104 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd3 key to auth repository 2020-03-11T22:44:52.104 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/ec-error-rollforward/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:44:52.595 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.3 2020-03-11T22:44:52.595 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.3 2020-03-11T22:44:52.595 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:44:52.596 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 3 --fsid=c7b6a6f8-a09d-4af9-b123-d357c2226ef8 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.53335/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:44:52.596 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:44:52.596 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:44:52.613 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:44:52.609+0000 7f8c097f7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:52.614 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:44:52.611+0000 7f8c097f7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:52.616 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:44:52.612+0000 7f8c097f7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:52.959 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 3 2020-03-11T22:44:52.959 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:44:52.959 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=3 2020-03-11T22:44:52.959 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:44:52.959 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:44:52.959 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:44:52.960 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:44:52.960 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:44:52.960 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:44:52.960 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:44:53.131 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:44:53.127+0000 7f8c097f7ec0 -1 Falling back to public interface 2020-03-11T22:44:53.334 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:44:53.878 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:44:53.874+0000 7f8c097f7ec0 -1 osd.3 0 log_to_monitors {default=true} 2020-03-11T22:44:54.336 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:44:54.409 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:44:54.409 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:44:54.410 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:44:54.410 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:44:54.410 INFO:tasks.workunit.client.0.smithi059.stdout:1 2020-03-11T22:44:54.729 INFO:tasks.workunit.client.0.smithi059.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/54837,v1:127.0.0.1:6827/54837] [v2:127.0.0.1:6828/54837,v1:127.0.0.1:6829/54837] exists,up b497abf9-591e-43e1-b78a-4ad9d9f2c595 2020-03-11T22:44:54.730 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:44:54.730 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:44:54.730 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:44:54.730 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:35: TEST_ec_error_rollforward: ceph osd erasure-code-profile set ec-profile m=2 k=2 crush-failure-domain=osd 2020-03-11T22:44:55.330 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:36: TEST_ec_error_rollforward: ceph osd pool create ec 1 1 erasure ec-profile 2020-03-11T22:44:56.028 INFO:tasks.workunit.client.0.smithi059.stderr:pool 'ec' already exists 2020-03-11T22:44:56.036 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:38: TEST_ec_error_rollforward: rados -p ec put foo /etc/passwd 2020-03-11T22:44:57.287 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:40: TEST_ec_error_rollforward: cat td/ec-error-rollforward/osd.2.pid 2020-03-11T22:44:57.288 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:40: TEST_ec_error_rollforward: kill -STOP 54482 2020-03-11T22:44:57.290 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:43: TEST_ec_error_rollforward: pids=55092 2020-03-11T22:44:57.290 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:44: TEST_ec_error_rollforward: sleep 1 2020-03-11T22:44:57.290 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:42: TEST_ec_error_rollforward: rados -p ec rm foo 2020-03-11T22:44:58.290 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:46: TEST_ec_error_rollforward: pids+=' 55115' 2020-03-11T22:44:58.290 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:45: TEST_ec_error_rollforward: rados -p ec rm a 2020-03-11T22:44:58.290 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:48: TEST_ec_error_rollforward: pids+=' 55116' 2020-03-11T22:44:58.291 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:47: TEST_ec_error_rollforward: rados -p ec rm b 2020-03-11T22:44:58.291 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:50: TEST_ec_error_rollforward: pids+=' 55117' 2020-03-11T22:44:58.291 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:51: TEST_ec_error_rollforward: sleep 1 2020-03-11T22:44:58.291 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:49: TEST_ec_error_rollforward: rados -p ec rm c 2020-03-11T22:44:59.291 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:54: TEST_ec_error_rollforward: kill_daemons td/ec-error-rollforward KILL osd 2020-03-11T22:44:59.292 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:44:59.292 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:44:59.293 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:44:59.293 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:44:59.293 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:44:59.402 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:44:59.402 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:55: TEST_ec_error_rollforward: kill 55092 55115 55116 55117 2020-03-11T22:44:59.402 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:56: TEST_ec_error_rollforward: wait 2020-03-11T22:44:59.407 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:58: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 0 2020-03-11T22:44:59.407 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/ec-error-rollforward 2020-03-11T22:44:59.407 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:44:59.408 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=0 2020-03-11T22:44:59.408 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:44:59.408 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/ec-error-rollforward/0 2020-03-11T22:44:59.408 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=c7b6a6f8-a09d-4af9-b123-d357c2226ef8 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-03-11T22:44:59.408 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:44:59.408 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:44:59.409 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:44:59.409 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/0' 2020-03-11T22:44:59.409 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/0/journal' 2020-03-11T22:44:59.409 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:44:59.409 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:44:59.409 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-03-11T22:44:59.410 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:44:59.410 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:44:59.410 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:44:59.410 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:44:59.410 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:44:59.410 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53335 2020-03-11T22:44:59.410 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53335/$cluster-$name.asok' 2020-03-11T22:44:59.411 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53335/$cluster-$name.asok' 2020-03-11T22:44:59.411 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:44:59.411 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-03-11T22:44:59.411 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-03-11T22:44:59.411 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:44:59.411 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:44:59.411 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:44:59.412 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:44:59.412 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:44:59.412 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/ec-error-rollforward/0 2020-03-11T22:44:59.412 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.0 2020-03-11T22:44:59.412 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 0 --fsid=c7b6a6f8-a09d-4af9-b123-d357c2226ef8 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.53335/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:44:59.412 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/ec-error-rollforward/0/whoami 2020-03-11T22:44:59.412 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.0 2020-03-11T22:44:59.413 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 0 = 0 ']' 2020-03-11T22:44:59.413 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:44:59.414 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:44:59.414 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:44:59.428 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:44:59.424+0000 7f99beb90ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:59.430 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:44:59.427+0000 7f99beb90ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:59.432 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:44:59.429+0000 7f99beb90ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:59.783 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 0 2020-03-11T22:44:59.784 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:44:59.784 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:44:59.784 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:44:59.784 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:44:59.784 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:44:59.784 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:44:59.784 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:44:59.785 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:44:59.785 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:44:59.948 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:44:59.944+0000 7f99beb90ec0 -1 Falling back to public interface 2020-03-11T22:45:00.162 INFO:tasks.workunit.client.0.smithi059.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/53764,v1:127.0.0.1:6803/53764] [v2:127.0.0.1:6804/53764,v1:127.0.0.1:6805/53764] exists,up 8355d9fe-0420-40f1-be0d-c6a59a7a9ced 2020-03-11T22:45:00.162 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.1 2020-03-11T22:45:00.163 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:45:00.163 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:45:00.163 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:45:00.163 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:59: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 1 2020-03-11T22:45:00.163 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/ec-error-rollforward 2020-03-11T22:45:00.163 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:45:00.164 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-11T22:45:00.164 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:45:00.164 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/ec-error-rollforward/1 2020-03-11T22:45:00.164 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=c7b6a6f8-a09d-4af9-b123-d357c2226ef8 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-03-11T22:45:00.164 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:45:00.165 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:45:00.165 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:45:00.165 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/1' 2020-03-11T22:45:00.165 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/1/journal' 2020-03-11T22:45:00.165 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:45:00.165 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:45:00.165 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-03-11T22:45:00.166 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:45:00.166 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:45:00.166 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:45:00.166 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:45:00.166 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:00.166 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53335 2020-03-11T22:45:00.166 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53335/$cluster-$name.asok' 2020-03-11T22:45:00.167 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53335/$cluster-$name.asok' 2020-03-11T22:45:00.167 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:45:00.167 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-03-11T22:45:00.167 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-03-11T22:45:00.167 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:45:00.167 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:45:00.167 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:45:00.168 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:45:00.168 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:45:00.168 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/ec-error-rollforward/1 2020-03-11T22:45:00.168 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-11T22:45:00.168 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=c7b6a6f8-a09d-4af9-b123-d357c2226ef8 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.53335/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:45:00.168 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/ec-error-rollforward/1/whoami 2020-03-11T22:45:00.168 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-11T22:45:00.169 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:45:00.169 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:45:00.169 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:45:00.179 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:45:00.175+0000 7f3af7c9fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:00.181 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:45:00.177+0000 7f3af7c9fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:00.183 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:45:00.179+0000 7f3af7c9fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:00.562 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 1 2020-03-11T22:45:00.562 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:45:00.562 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:45:00.562 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:45:00.562 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:45:00.563 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:45:00.563 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:45:00.563 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:45:00.563 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:45:00.563 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:45:00.698 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:45:00.694+0000 7f3af7c9fec0 -1 Falling back to public interface 2020-03-11T22:45:00.928 INFO:tasks.workunit.client.0.smithi059.stdout:osd.1 up in weight 1 up_from 10 up_thru 24 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/54127,v1:127.0.0.1:6811/54127] [v2:127.0.0.1:6812/54127,v1:127.0.0.1:6813/54127] exists,up a9e70d5d-6bd3-4537-8695-a71a2a70af22 2020-03-11T22:45:00.928 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.2 2020-03-11T22:45:00.929 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:45:00.929 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:45:00.929 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:45:00.929 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:60: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 2 2020-03-11T22:45:00.929 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/ec-error-rollforward 2020-03-11T22:45:00.929 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:45:00.930 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=2 2020-03-11T22:45:00.930 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:45:00.930 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/ec-error-rollforward/2 2020-03-11T22:45:00.930 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=c7b6a6f8-a09d-4af9-b123-d357c2226ef8 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-03-11T22:45:00.930 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:45:00.930 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:45:00.930 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:45:00.931 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/2' 2020-03-11T22:45:00.931 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/2/journal' 2020-03-11T22:45:00.931 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:45:00.931 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:45:00.931 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-03-11T22:45:00.931 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:45:00.931 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:45:00.932 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:45:00.932 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:45:00.932 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:00.932 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53335 2020-03-11T22:45:00.932 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53335/$cluster-$name.asok' 2020-03-11T22:45:00.932 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53335/$cluster-$name.asok' 2020-03-11T22:45:00.932 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:45:00.933 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-03-11T22:45:00.933 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-03-11T22:45:00.933 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:45:00.933 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:45:00.933 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:45:00.933 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:45:00.933 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:45:00.934 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/ec-error-rollforward/2 2020-03-11T22:45:00.934 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.2 2020-03-11T22:45:00.934 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 2 --fsid=c7b6a6f8-a09d-4af9-b123-d357c2226ef8 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.53335/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:45:00.934 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/ec-error-rollforward/2/whoami 2020-03-11T22:45:00.934 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 2 = 2 ']' 2020-03-11T22:45:00.934 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:45:00.934 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:45:00.935 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:45:00.935 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:45:00.932+0000 7f99beb90ec0 -1 osd.0 26 log_to_monitors {default=true} 2020-03-11T22:45:00.946 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:45:00.943+0000 7f71086afec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:00.948 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:45:00.945+0000 7f71086afec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:00.950 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:45:00.946+0000 7f71086afec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:01.298 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 2 2020-03-11T22:45:01.298 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:45:01.298 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:45:01.298 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:45:01.299 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:45:01.299 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:45:01.299 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:45:01.299 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:45:01.299 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:45:01.299 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:45:01.466 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:45:01.463+0000 7f71086afec0 -1 Falling back to public interface 2020-03-11T22:45:01.720 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:45:01.838 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:45:01.835+0000 7f3af7c9fec0 -1 osd.1 26 log_to_monitors {default=true} 2020-03-11T22:45:02.716 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:45:02.713+0000 7f71086afec0 -1 osd.2 25 log_to_monitors {default=true} 2020-03-11T22:45:02.721 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:45:02.722 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:45:02.722 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:45:02.722 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:45:02.722 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:45:02.722 INFO:tasks.workunit.client.0.smithi059.stdout:1 2020-03-11T22:45:03.122 INFO:tasks.workunit.client.0.smithi059.stdout:osd.2 up in weight 1 up_from 33 up_thru 0 down_at 29 last_clean_interval [15,25) [v2:127.0.0.1:6818/55471,v1:127.0.0.1:6819/55471] [v2:127.0.0.1:6820/55471,v1:127.0.0.1:6821/55471] exists,up 0e421bdb-2cd6-4049-b2dc-2be5fd0855e0 2020-03-11T22:45:03.123 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.3 2020-03-11T22:45:03.123 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:45:03.123 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:45:03.123 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:45:03.123 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:61: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 3 2020-03-11T22:45:03.124 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/ec-error-rollforward 2020-03-11T22:45:03.124 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:45:03.124 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=3 2020-03-11T22:45:03.124 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:45:03.124 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/ec-error-rollforward/3 2020-03-11T22:45:03.124 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=c7b6a6f8-a09d-4af9-b123-d357c2226ef8 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-03-11T22:45:03.125 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:45:03.125 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:45:03.125 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:45:03.125 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/3' 2020-03-11T22:45:03.125 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/3/journal' 2020-03-11T22:45:03.126 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:45:03.126 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:45:03.126 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-03-11T22:45:03.126 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:45:03.126 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:45:03.126 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:45:03.126 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:45:03.127 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:03.127 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53335 2020-03-11T22:45:03.127 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53335/$cluster-$name.asok' 2020-03-11T22:45:03.127 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53335/$cluster-$name.asok' 2020-03-11T22:45:03.127 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:45:03.127 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-03-11T22:45:03.127 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-03-11T22:45:03.128 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:45:03.128 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:45:03.128 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:45:03.128 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:45:03.128 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:45:03.128 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/ec-error-rollforward/3 2020-03-11T22:45:03.128 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.3 2020-03-11T22:45:03.129 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 3 --fsid=c7b6a6f8-a09d-4af9-b123-d357c2226ef8 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.53335/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:45:03.129 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/ec-error-rollforward/3/whoami 2020-03-11T22:45:03.129 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 3 = 3 ']' 2020-03-11T22:45:03.130 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:45:03.130 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:45:03.130 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:45:03.141 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:45:03.138+0000 7feac156bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:03.165 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:45:03.139+0000 7feac156bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:03.166 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:45:03.163+0000 7feac156bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:03.542 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 3 2020-03-11T22:45:03.542 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:45:03.543 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=3 2020-03-11T22:45:03.543 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:45:03.543 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:45:03.543 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:45:03.543 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:45:03.543 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:45:03.544 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:45:03.544 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:45:03.686 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:45:03.682+0000 7feac156bec0 -1 Falling back to public interface 2020-03-11T22:45:03.902 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:45:04.611 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:45:04.607+0000 7feac156bec0 -1 osd.3 26 log_to_monitors {default=true} 2020-03-11T22:45:04.903 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:45:04.903 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:45:04.904 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:45:04.904 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:45:04.904 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:45:04.904 INFO:tasks.workunit.client.0.smithi059.stdout:1 2020-03-11T22:45:05.298 INFO:tasks.workunit.client.0.smithi059.stdout:osd.3 up in weight 1 up_from 35 up_thru 0 down_at 29 last_clean_interval [20,26) [v2:127.0.0.1:6826/55748,v1:127.0.0.1:6827/55748] [v2:127.0.0.1:6828/55748,v1:127.0.0.1:6829/55748] exists,up b497abf9-591e-43e1-b78a-4ad9d9f2c595 2020-03-11T22:45:05.298 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:45:05.298 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:45:05.299 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:45:05.299 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:63: TEST_ec_error_rollforward: wait_for_clean 2020-03-11T22:45:05.299 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:45:05.299 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:45:05.299 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:45:05.299 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:45:05.300 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:45:05.300 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:45:05.300 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:45:05.300 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:45:05.301 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:45:05.301 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:45:05.391 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:45:05.391 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:45:05.391 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:45:05.391 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:45:05.392 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:45:05.757 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:45:05.757 INFO:tasks.workunit.client.0.smithi059.stderr:1 2020-03-11T22:45:05.758 INFO:tasks.workunit.client.0.smithi059.stderr:2 2020-03-11T22:45:05.758 INFO:tasks.workunit.client.0.smithi059.stderr:3' 2020-03-11T22:45:05.758 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:45:05.758 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:45:05.758 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:45:05.915 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=120259084291 2020-03-11T22:45:05.915 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-120259084291' 2020-03-11T22:45:05.915 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:45:05.916 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:45:06.042 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=133143986179 2020-03-11T22:45:06.043 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-120259084291 1-133143986179' 2020-03-11T22:45:06.043 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:45:06.043 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:45:06.169 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=141733920771 2020-03-11T22:45:06.169 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-120259084291 1-133143986179 2-141733920771' 2020-03-11T22:45:06.169 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:45:06.170 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:45:06.296 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=150323855363 2020-03-11T22:45:06.296 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-120259084291 1-133143986179 2-141733920771 3-150323855363' 2020-03-11T22:45:06.296 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:45:06.297 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-120259084291 2020-03-11T22:45:06.297 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:45:06.298 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:45:06.299 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-120259084291 2020-03-11T22:45:06.299 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:45:06.300 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=120259084291 2020-03-11T22:45:06.300 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 120259084291' 2020-03-11T22:45:06.300 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.0 seq 120259084291 2020-03-11T22:45:06.301 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:45:06.660 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 120259084292 -lt 120259084291 2020-03-11T22:45:06.660 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:45:06.661 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-133143986179 2020-03-11T22:45:06.661 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:45:06.662 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:45:06.663 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-133143986179 2020-03-11T22:45:06.663 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:45:06.664 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=133143986179 2020-03-11T22:45:06.664 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 133143986179' 2020-03-11T22:45:06.664 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.1 seq 133143986179 2020-03-11T22:45:06.665 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:45:07.052 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 133143986179 -lt 133143986179 2020-03-11T22:45:07.052 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:45:07.053 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-141733920771 2020-03-11T22:45:07.053 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:45:07.054 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:45:07.055 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-141733920771 2020-03-11T22:45:07.055 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:45:07.056 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=141733920771 2020-03-11T22:45:07.056 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 141733920771' 2020-03-11T22:45:07.057 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.2 seq 141733920771 2020-03-11T22:45:07.057 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:45:07.416 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 141733920771 -lt 141733920771 2020-03-11T22:45:07.416 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:45:07.417 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-150323855363 2020-03-11T22:45:07.417 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:45:07.418 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:45:07.419 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-150323855363 2020-03-11T22:45:07.419 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:45:07.420 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=150323855363 2020-03-11T22:45:07.420 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 150323855363' 2020-03-11T22:45:07.421 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.3 seq 150323855363 2020-03-11T22:45:07.421 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:45:07.783 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 150323855363 -lt 150323855363 2020-03-11T22:45:07.784 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:45:07.784 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:45:07.784 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:45:08.309 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:45:08.309 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:45:08.310 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:45:08.310 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:45:08.310 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:45:08.310 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:45:08.311 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:45:08.311 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:45:08.656 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:45:08.657 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:45:08.657 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:45:08.657 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:45:09.127 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:45:09.127 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:45:09.127 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:45:09.127 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:22: run: teardown td/ec-error-rollforward 2020-03-11T22:45:09.127 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2020-03-11T22:45:09.127 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:45:09.127 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2020-03-11T22:45:09.128 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:45:09.128 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:45:09.129 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:45:09.129 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:45:09.129 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:45:09.236 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:45:09.237 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:45:09.238 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:45:09.238 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:45:09.238 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:45:09.238 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:45:09.239 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:45:09.240 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:45:09.240 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:45:09.241 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:45:09.241 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:45:09.242 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:45:09.242 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:45:09.243 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2020-03-11T22:45:09.248 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:45:09.248 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:09.249 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53335 2020-03-11T22:45:09.249 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.53335 2020-03-11T22:45:09.250 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:45:09.250 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:45:09.251 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: code=0 2020-03-11T22:45:09.251 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: teardown td/ec-error-rollforward 0 2020-03-11T22:45:09.251 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2020-03-11T22:45:09.251 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2020-03-11T22:45:09.251 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2020-03-11T22:45:09.251 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:45:09.252 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:45:09.252 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:45:09.252 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:45:09.252 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:45:09.253 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:45:09.253 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:45:09.253 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:45:09.253 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:45:09.254 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:45:09.254 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:45:09.254 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:45:09.255 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:45:09.255 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:45:09.256 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:45:09.256 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:45:09.257 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:45:09.257 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2020-03-11T22:45:09.258 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2020-03-11T22:45:09.258 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:45:09.259 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:09.259 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53335 2020-03-11T22:45:09.259 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.53335 2020-03-11T22:45:09.260 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:45:09.260 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:45:09.260 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: return 0 2020-03-11T22:45:09.261 INFO:teuthology.orchestra.run:Running command with timeout 3600 2020-03-11T22:45:09.261 INFO:teuthology.orchestra.run.smithi059:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2020-03-11T22:45:09.336 INFO:tasks.workunit:Running workunit osd/osd-backfill-prio.sh... 2020-03-11T22:45:09.336 INFO:teuthology.orchestra.run.smithi059:workunit test osd/osd-backfill-prio.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4acc1121e47039a3cdf2978ae458c10d153aa141 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh 2020-03-11T22:45:09.392 INFO:tasks.workunit.client.0.smithi059.stderr:stty: 'standard input': Inappropriate ioctl for device 2020-03-11T22:45:09.401 INFO:tasks.workunit.client.0.smithi059.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2020-03-11T22:45:09.402 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2020-03-11T22:45:09.402 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2020-03-11T22:45:09.402 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: main: export PYTHONWARNINGS=ignore 2020-03-11T22:45:09.402 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: main: PYTHONWARNINGS=ignore 2020-03-11T22:45:09.402 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: main: export CEPH_CONF=/dev/null 2020-03-11T22:45:09.402 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: main: CEPH_CONF=/dev/null 2020-03-11T22:45:09.403 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: main: unset CEPH_ARGS 2020-03-11T22:45:09.403 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: local code 2020-03-11T22:45:09.403 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: run td/osd-backfill-prio 2020-03-11T22:45:09.403 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:21: run: local dir=td/osd-backfill-prio 2020-03-11T22:45:09.403 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:22: run: shift 2020-03-11T22:45:09.403 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:25: run: export CEPH_MON=127.0.0.1:7114 2020-03-11T22:45:09.403 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:25: run: CEPH_MON=127.0.0.1:7114 2020-03-11T22:45:09.404 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:26: run: export CEPH_ARGS 2020-03-11T22:45:09.404 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:27: run: uuidgen 2020-03-11T22:45:09.404 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:27: run: CEPH_ARGS+='--fsid=34f3141b-0483-4f39-a129-ea18fdfbfb6a --auth-supported=none ' 2020-03-11T22:45:09.404 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:45:09.404 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:29: run: CEPH_ARGS+='--osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-03-11T22:45:09.404 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:30: run: export objects=50 2020-03-11T22:45:09.404 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:30: run: objects=50 2020-03-11T22:45:09.405 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:31: run: export poolprefix=test 2020-03-11T22:45:09.405 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:31: run: poolprefix=test 2020-03-11T22:45:09.405 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:32: run: export FORCE_PRIO=254 2020-03-11T22:45:09.405 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:32: run: FORCE_PRIO=254 2020-03-11T22:45:09.405 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:33: run: export DEGRADED_PRIO=150 2020-03-11T22:45:09.405 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:33: run: DEGRADED_PRIO=150 2020-03-11T22:45:09.405 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:34: run: export NORMAL_PRIO=110 2020-03-11T22:45:09.406 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:34: run: NORMAL_PRIO=110 2020-03-11T22:45:09.406 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:36: run: set 2020-03-11T22:45:09.406 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:36: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2020-03-11T22:45:09.408 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:36: run: local 'funcs=TEST_backfill_pool_priority 2020-03-11T22:45:09.409 INFO:tasks.workunit.client.0.smithi059.stderr:TEST_backfill_priority' 2020-03-11T22:45:09.409 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:37: run: for func in $funcs 2020-03-11T22:45:09.409 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:38: run: setup td/osd-backfill-prio 2020-03-11T22:45:09.409 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-prio 2020-03-11T22:45:09.409 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-prio 2020-03-11T22:45:09.409 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-prio 2020-03-11T22:45:09.410 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:45:09.410 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-prio KILL 2020-03-11T22:45:09.410 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:45:09.410 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:45:09.410 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:45:09.410 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:45:09.410 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:45:09.411 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:45:09.411 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:45:09.411 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:45:09.411 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:45:09.411 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:45:09.411 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:45:09.411 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:45:09.411 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:45:09.412 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:45:09.412 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:45:09.412 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:45:09.413 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:45:09.414 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:45:09.414 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-prio 2020-03-11T22:45:09.414 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:45:09.414 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:09.414 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56462 2020-03-11T22:45:09.414 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.56462 2020-03-11T22:45:09.414 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:45:09.415 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:45:09.415 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-prio 2020-03-11T22:45:09.415 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-11T22:45:09.415 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:09.415 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56462 2020-03-11T22:45:09.415 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.56462 2020-03-11T22:45:09.417 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-11T22:45:09.417 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2020-03-11T22:45:09.418 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2020-03-11T22:45:09.418 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-11T22:45:09.418 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-prio 1' TERM HUP INT 2020-03-11T22:45:09.418 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:39: run: TEST_backfill_pool_priority td/osd-backfill-prio 2020-03-11T22:45:09.418 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:326: TEST_backfill_pool_priority: local dir=td/osd-backfill-prio 2020-03-11T22:45:09.418 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:327: TEST_backfill_pool_priority: local pools=3 2020-03-11T22:45:09.418 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:328: TEST_backfill_pool_priority: local OSDS=2 2020-03-11T22:45:09.419 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:330: TEST_backfill_pool_priority: run_mon td/osd-backfill-prio a 2020-03-11T22:45:09.419 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-prio 2020-03-11T22:45:09.419 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-11T22:45:09.419 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-11T22:45:09.419 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-11T22:45:09.419 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-prio/a 2020-03-11T22:45:09.419 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-prio/a --run-dir=td/osd-backfill-prio 2020-03-11T22:45:09.714 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-11T22:45:09.714 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:45:09.714 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:45:09.714 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:45:09.715 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:09.715 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56462 2020-03-11T22:45:09.715 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.56462/$cluster-$name.asok' 2020-03-11T22:45:09.715 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-prio/a '--log-file=td/osd-backfill-prio/$name.log' '--admin-socket=/tmp/ceph-asok.56462/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-prio/log --run-dir=td/osd-backfill-prio '--pid-file=td/osd-backfill-prio/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-11T22:45:09.873 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-11T22:45:09.875 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-11T22:45:09.875 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:45:09.875 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:45:09.876 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-11T22:45:09.876 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-11T22:45:09.876 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:45:09.876 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:45:09.876 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:45:09.876 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:45:09.876 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:09.877 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56462 2020-03-11T22:45:09.877 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56462/ceph-mon.a.asok 2020-03-11T22:45:09.877 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:45:09.877 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.56462/ceph-mon.a.asok config get fsid 2020-03-11T22:45:09.968 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-11T22:45:09.968 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:45:09.968 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:45:09.968 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-11T22:45:09.969 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-11T22:45:09.969 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:45:09.969 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:45:09.969 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:45:09.969 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:45:09.969 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:09.970 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56462 2020-03-11T22:45:09.970 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56462/ceph-mon.a.asok 2020-03-11T22:45:09.970 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:45:09.970 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.56462/ceph-mon.a.asok config get mon_host 2020-03-11T22:45:10.058 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:331: TEST_backfill_pool_priority: run_mgr td/osd-backfill-prio x 2020-03-11T22:45:10.058 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/osd-backfill-prio 2020-03-11T22:45:10.058 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-11T22:45:10.058 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-11T22:45:10.058 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-11T22:45:10.059 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/osd-backfill-prio/x 2020-03-11T22:45:10.059 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-11T22:45:10.959 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-11T22:45:10.959 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:45:10.959 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:45:10.959 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:45:10.960 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:10.960 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56462 2020-03-11T22:45:10.960 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.56462/$cluster-$name.asok' 2020-03-11T22:45:10.960 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:45:10.960 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-prio/x '--log-file=td/osd-backfill-prio/$name.log' '--admin-socket=/tmp/ceph-asok.56462/$cluster-$name.asok' --run-dir=td/osd-backfill-prio '--pid-file=td/osd-backfill-prio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:45:10.984 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:332: TEST_backfill_pool_priority: export CEPH_ARGS 2020-03-11T22:45:10.984 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:334: TEST_backfill_pool_priority: expr 2 - 1 2020-03-11T22:45:10.985 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:334: TEST_backfill_pool_priority: seq 0 1 2020-03-11T22:45:10.989 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:334: TEST_backfill_pool_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:45:10.990 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:336: TEST_backfill_pool_priority: run_osd td/osd-backfill-prio 0 2020-03-11T22:45:10.990 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-prio 2020-03-11T22:45:10.990 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:45:10.990 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-11T22:45:10.990 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:45:10.990 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-prio/0 2020-03-11T22:45:10.990 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=34f3141b-0483-4f39-a129-ea18fdfbfb6a --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-03-11T22:45:10.991 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:45:10.991 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:45:10.991 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:45:10.991 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/0' 2020-03-11T22:45:10.991 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/0/journal' 2020-03-11T22:45:10.991 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:45:10.991 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:45:10.991 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2020-03-11T22:45:10.992 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:45:10.992 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:45:10.992 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:45:10.992 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:45:10.992 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:10.992 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56462 2020-03-11T22:45:10.992 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.56462/$cluster-$name.asok' 2020-03-11T22:45:10.993 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56462/$cluster-$name.asok' 2020-03-11T22:45:10.993 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:45:10.993 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:45:10.993 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:45:10.993 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2020-03-11T22:45:10.993 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2020-03-11T22:45:10.994 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:45:10.994 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:45:10.994 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:45:10.994 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:45:10.994 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:45:10.994 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-prio/0 2020-03-11T22:45:10.995 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:45:10.995 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=a177027d-b62f-494d-b8f1-9a8ea4a6998a 2020-03-11T22:45:10.995 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 a177027d-b62f-494d-b8f1-9a8ea4a6998a' 2020-03-11T22:45:10.995 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:45:10.995 INFO:tasks.workunit.client.0.smithi059.stdout:add osd0 a177027d-b62f-494d-b8f1-9a8ea4a6998a 2020-03-11T22:45:11.006 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQD3aWleJUI2ABAADknC0ztziEL+2uR62WziZg== 2020-03-11T22:45:11.006 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQD3aWleJUI2ABAADknC0ztziEL+2uR62WziZg=="}' 2020-03-11T22:45:11.006 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new a177027d-b62f-494d-b8f1-9a8ea4a6998a -i td/osd-backfill-prio/0/new.json 2020-03-11T22:45:11.317 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:45:11.326 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-prio/0/new.json 2020-03-11T22:45:11.327 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=34f3141b-0483-4f39-a129-ea18fdfbfb6a --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/0 --osd-journal=td/osd-backfill-prio/0/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56462/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQD3aWleJUI2ABAADknC0ztziEL+2uR62WziZg== --osd-uuid a177027d-b62f-494d-b8f1-9a8ea4a6998a 2020-03-11T22:45:11.355 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:45:11.352+0000 7fe45d13eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:11.356 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:45:11.354+0000 7fe45d13eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:11.358 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:45:11.355+0000 7fe45d13eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:11.359 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:45:11.356+0000 7fe45d13eec0 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2020-03-11T22:45:11.359 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:45:11.356+0000 7fe45d13eec0 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2020-03-11T22:45:11.359 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:45:11.356+0000 7fe45d13eec0 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2020-03-11T22:45:11.448 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:45:11.445+0000 7fe45d13eec0 -1 bluestore(td/osd-backfill-prio/0) _read_fsid unparsable uuid 2020-03-11T22:45:14.548 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-prio/0/keyring 2020-03-11T22:45:14.548 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:45:14.549 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-11T22:45:14.549 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-prio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:45:14.549 INFO:tasks.workunit.client.0.smithi059.stdout:adding osd0 key to auth repository 2020-03-11T22:45:15.075 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-11T22:45:15.076 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=34f3141b-0483-4f39-a129-ea18fdfbfb6a --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/0 --osd-journal=td/osd-backfill-prio/0/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56462/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:45:15.076 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:45:15.076 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:45:15.077 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:45:15.077 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.0 2020-03-11T22:45:15.099 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:45:15.097+0000 7f0df2cb2ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:15.101 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:45:15.098+0000 7f0df2cb2ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:15.102 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:45:15.100+0000 7f0df2cb2ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:15.439 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-11T22:45:15.439 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:45:15.439 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:45:15.440 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:45:15.440 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:45:15.440 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:45:15.440 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:45:15.440 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:45:15.440 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:45:15.441 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:45:15.618 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:45:15.616+0000 7f0df2cb2ec0 -1 Falling back to public interface 2020-03-11T22:45:15.835 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:45:16.363 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:45:16.361+0000 7f0df2cb2ec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-11T22:45:16.836 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:45:16.837 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:45:16.837 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:45:16.837 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:45:16.837 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:45:16.837 INFO:tasks.workunit.client.0.smithi059.stdout:1 2020-03-11T22:45:17.226 INFO:tasks.workunit.client.0.smithi059.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/56894,v1:127.0.0.1:6803/56894] [v2:127.0.0.1:6804/56894,v1:127.0.0.1:6805/56894] exists,up a177027d-b62f-494d-b8f1-9a8ea4a6998a 2020-03-11T22:45:17.226 INFO:tasks.workunit.client.0.smithi059.stdout:add osd1 84430dbe-6e41-43af-908a-8f848f4205ea 2020-03-11T22:45:17.227 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:45:17.227 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:45:17.227 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:45:17.227 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:334: TEST_backfill_pool_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:45:17.227 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:336: TEST_backfill_pool_priority: run_osd td/osd-backfill-prio 1 2020-03-11T22:45:17.227 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-prio 2020-03-11T22:45:17.227 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:45:17.228 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-11T22:45:17.228 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:45:17.228 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-prio/1 2020-03-11T22:45:17.228 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=34f3141b-0483-4f39-a129-ea18fdfbfb6a --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-03-11T22:45:17.228 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:45:17.228 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:45:17.228 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:45:17.229 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/1' 2020-03-11T22:45:17.229 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/1/journal' 2020-03-11T22:45:17.229 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:45:17.229 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:45:17.229 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2020-03-11T22:45:17.229 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:45:17.229 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:45:17.229 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:45:17.230 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:45:17.230 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:17.230 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56462 2020-03-11T22:45:17.230 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.56462/$cluster-$name.asok' 2020-03-11T22:45:17.230 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56462/$cluster-$name.asok' 2020-03-11T22:45:17.230 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:45:17.230 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:45:17.231 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:45:17.231 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2020-03-11T22:45:17.231 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2020-03-11T22:45:17.231 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:45:17.231 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:45:17.231 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:45:17.231 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:45:17.231 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:45:17.232 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-prio/1 2020-03-11T22:45:17.232 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:45:17.232 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=84430dbe-6e41-43af-908a-8f848f4205ea 2020-03-11T22:45:17.232 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 84430dbe-6e41-43af-908a-8f848f4205ea' 2020-03-11T22:45:17.232 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:45:17.242 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQD9aWleYNxNDhAAdjDbdULz+SYI46WW+BD6MQ== 2020-03-11T22:45:17.242 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQD9aWleYNxNDhAAdjDbdULz+SYI46WW+BD6MQ=="}' 2020-03-11T22:45:17.243 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 84430dbe-6e41-43af-908a-8f848f4205ea -i td/osd-backfill-prio/1/new.json 2020-03-11T22:45:17.737 INFO:tasks.workunit.client.0.smithi059.stdout:1 2020-03-11T22:45:17.747 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-prio/1/new.json 2020-03-11T22:45:17.747 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=34f3141b-0483-4f39-a129-ea18fdfbfb6a --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/1 --osd-journal=td/osd-backfill-prio/1/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56462/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQD9aWleYNxNDhAAdjDbdULz+SYI46WW+BD6MQ== --osd-uuid 84430dbe-6e41-43af-908a-8f848f4205ea 2020-03-11T22:45:17.774 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:45:17.772+0000 7f2424a40ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:17.778 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:45:17.776+0000 7f2424a40ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:17.780 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:45:17.778+0000 7f2424a40ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:17.781 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:45:17.778+0000 7f2424a40ec0 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2020-03-11T22:45:17.781 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:45:17.778+0000 7f2424a40ec0 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2020-03-11T22:45:17.781 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:45:17.778+0000 7f2424a40ec0 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2020-03-11T22:45:17.878 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:45:17.876+0000 7f2424a40ec0 -1 bluestore(td/osd-backfill-prio/1) _read_fsid unparsable uuid 2020-03-11T22:45:20.953 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-prio/1/keyring 2020-03-11T22:45:20.954 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:45:20.955 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-03-11T22:45:20.955 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-prio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:45:20.955 INFO:tasks.workunit.client.0.smithi059.stdout:adding osd1 key to auth repository 2020-03-11T22:45:21.479 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.1 2020-03-11T22:45:21.479 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-03-11T22:45:21.479 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:45:21.479 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=34f3141b-0483-4f39-a129-ea18fdfbfb6a --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/1 --osd-journal=td/osd-backfill-prio/1/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56462/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:45:21.480 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:45:21.480 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:45:21.502 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:45:21.487+0000 7f5b909f9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:21.503 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:45:21.489+0000 7f5b909f9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:21.504 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:45:21.490+0000 7f5b909f9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:21.857 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-03-11T22:45:21.857 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:45:21.857 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:45:21.857 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:45:21.857 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:45:21.858 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:45:21.858 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:45:21.858 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:45:21.858 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:45:21.858 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:45:22.009 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:45:22.005+0000 7f5b909f9ec0 -1 Falling back to public interface 2020-03-11T22:45:22.236 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:45:22.819 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:45:22.817+0000 7f5b909f9ec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-11T22:45:23.237 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:45:23.238 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:45:23.238 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:45:23.238 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:45:23.238 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:45:23.238 INFO:tasks.workunit.client.0.smithi059.stdout:1 2020-03-11T22:45:23.685 INFO:tasks.workunit.client.0.smithi059.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/57249,v1:127.0.0.1:6811/57249] [v2:127.0.0.1:6812/57249,v1:127.0.0.1:6813/57249] exists,up 84430dbe-6e41-43af-908a-8f848f4205ea 2020-03-11T22:45:23.685 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:45:23.685 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:45:23.686 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:45:23.686 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:339: TEST_backfill_pool_priority: seq 1 3 2020-03-11T22:45:23.687 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:339: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-03-11T22:45:23.687 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:341: TEST_backfill_pool_priority: create_pool test1 1 1 2020-03-11T22:45:23.687 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test1 1 1 2020-03-11T22:45:24.253 INFO:tasks.workunit.client.0.smithi059.stderr:pool 'test1' already exists 2020-03-11T22:45:24.261 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:45:25.262 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:342: TEST_backfill_pool_priority: ceph osd pool set test1 size 2 2020-03-11T22:45:26.091 INFO:tasks.workunit.client.0.smithi059.stderr:set pool 1 size to 2 2020-03-11T22:45:26.100 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:339: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-03-11T22:45:26.100 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:341: TEST_backfill_pool_priority: create_pool test2 1 1 2020-03-11T22:45:26.101 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test2 1 1 2020-03-11T22:45:26.735 INFO:tasks.workunit.client.0.smithi059.stderr:pool 'test2' already exists 2020-03-11T22:45:26.742 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:45:27.743 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:342: TEST_backfill_pool_priority: ceph osd pool set test2 size 2 2020-03-11T22:45:28.562 INFO:tasks.workunit.client.0.smithi059.stderr:set pool 2 size to 2 2020-03-11T22:45:28.570 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:339: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-03-11T22:45:28.570 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:341: TEST_backfill_pool_priority: create_pool test3 1 1 2020-03-11T22:45:28.571 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test3 1 1 2020-03-11T22:45:29.092 INFO:tasks.workunit.client.0.smithi059.stderr:pool 'test3' already exists 2020-03-11T22:45:29.100 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:45:30.101 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:342: TEST_backfill_pool_priority: ceph osd pool set test3 size 2 2020-03-11T22:45:30.874 INFO:tasks.workunit.client.0.smithi059.stderr:set pool 3 size to 2 2020-03-11T22:45:30.882 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:344: TEST_backfill_pool_priority: sleep 5 2020-03-11T22:45:35.884 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:346: TEST_backfill_pool_priority: wait_for_clean 2020-03-11T22:45:35.884 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:45:35.884 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:45:35.885 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:45:35.885 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:45:35.886 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:45:35.886 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:45:35.886 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:45:35.886 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:45:35.886 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:45:35.886 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:45:35.991 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:45:35.992 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:45:35.992 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:45:35.992 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:45:35.992 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:45:36.363 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:45:36.364 INFO:tasks.workunit.client.0.smithi059.stderr:1' 2020-03-11T22:45:36.364 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:45:36.364 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:45:36.364 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:45:36.497 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836486 2020-03-11T22:45:36.497 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486' 2020-03-11T22:45:36.497 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:45:36.498 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:45:36.631 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672965 2020-03-11T22:45:36.631 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486 1-42949672965' 2020-03-11T22:45:36.631 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:45:36.632 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836486 2020-03-11T22:45:36.632 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:45:36.632 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:45:36.634 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836486 2020-03-11T22:45:36.634 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:45:36.635 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836486 2020-03-11T22:45:36.635 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2020-03-11T22:45:36.635 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.0 seq 21474836486 2020-03-11T22:45:36.635 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:45:37.004 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836484 -lt 21474836486 2020-03-11T22:45:37.004 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:45:38.005 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:45:38.006 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:45:38.383 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836484 -lt 21474836486 2020-03-11T22:45:38.383 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:45:39.384 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 299 -eq 0 ']' 2020-03-11T22:45:39.385 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:45:39.763 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836487 -lt 21474836486 2020-03-11T22:45:39.763 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:45:39.764 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672965 2020-03-11T22:45:39.764 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:45:39.765 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:45:39.766 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672965 2020-03-11T22:45:39.766 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:45:39.767 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672965 2020-03-11T22:45:39.767 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2020-03-11T22:45:39.767 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.1 seq 42949672965 2020-03-11T22:45:39.768 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:45:40.136 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672965 -lt 42949672965 2020-03-11T22:45:40.137 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:45:40.137 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:45:40.137 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:45:40.616 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 3 == 0 2020-03-11T22:45:40.616 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:45:40.617 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:45:40.617 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:45:40.617 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:45:40.618 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:45:40.618 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:45:40.618 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:45:40.985 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=3 2020-03-11T22:45:40.986 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:45:40.986 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:45:40.986 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:45:41.466 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 3 = 3 2020-03-11T22:45:41.466 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:45:41.467 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:45:41.467 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:348: TEST_backfill_pool_priority: ceph pg dump pgs 2020-03-11T22:45:41.814 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:45:41.815 INFO:tasks.workunit.client.0.smithi059.stdout:3.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:45:31.525902+0000 0'0 28:18 [1,0] 1 [1,0] 1 0'0 2020-03-11T22:45:28.906011+0000 0'0 2020-03-11T22:45:28.906011+0000 0 2020-03-11T22:45:41.815 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:45:25.926761+0000 0'0 27:28 [1,0] 1 [1,0] 1 0'0 2020-03-11T22:45:24.081865+0000 0'0 2020-03-11T22:45:24.081865+0000 0 2020-03-11T22:45:41.815 INFO:tasks.workunit.client.0.smithi059.stdout:2.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:45:28.639930+0000 0'0 26:23 [0,1] 0 [0,1] 0 0'0 2020-03-11T22:45:26.454442+0000 0'0 2020-03-11T22:45:26.454442+0000 0 2020-03-11T22:45:41.815 INFO:tasks.workunit.client.0.smithi059.stdout: 2020-03-11T22:45:41.815 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:45:41.816 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2020-03-11T22:45:41.825 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:352: TEST_backfill_pool_priority: local PG1 2020-03-11T22:45:41.825 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:353: TEST_backfill_pool_priority: local POOLNUM1 2020-03-11T22:45:41.825 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:354: TEST_backfill_pool_priority: local pool1 2020-03-11T22:45:41.825 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:355: TEST_backfill_pool_priority: local chk_osd1_1 2020-03-11T22:45:41.825 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:356: TEST_backfill_pool_priority: local chk_osd1_2 2020-03-11T22:45:41.826 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:358: TEST_backfill_pool_priority: local PG2 2020-03-11T22:45:41.826 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:359: TEST_backfill_pool_priority: local POOLNUM2 2020-03-11T22:45:41.826 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:360: TEST_backfill_pool_priority: local pool2 2020-03-11T22:45:41.826 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:361: TEST_backfill_pool_priority: local chk_osd2_1 2020-03-11T22:45:41.826 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:362: TEST_backfill_pool_priority: local chk_osd2_2 2020-03-11T22:45:41.827 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:364: TEST_backfill_pool_priority: seq 1 3 2020-03-11T22:45:41.827 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:364: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-03-11T22:45:41.827 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:366: TEST_backfill_pool_priority: ceph pg map 1.0 --format=json 2020-03-11T22:45:41.827 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:366: TEST_backfill_pool_priority: jq '.acting[]' 2020-03-11T22:45:42.184 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:367: TEST_backfill_pool_priority: head -1 td/osd-backfill-prio/acting 2020-03-11T22:45:42.185 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:367: TEST_backfill_pool_priority: local test_osd1=1 2020-03-11T22:45:42.186 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:368: TEST_backfill_pool_priority: tail -1 td/osd-backfill-prio/acting 2020-03-11T22:45:42.187 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:368: TEST_backfill_pool_priority: local test_osd2=0 2020-03-11T22:45:42.187 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:369: TEST_backfill_pool_priority: '[' -z '' ']' 2020-03-11T22:45:42.187 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:371: TEST_backfill_pool_priority: PG1=1.0 2020-03-11T22:45:42.188 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:372: TEST_backfill_pool_priority: POOLNUM1=1 2020-03-11T22:45:42.188 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:373: TEST_backfill_pool_priority: pool1=test1 2020-03-11T22:45:42.188 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:374: TEST_backfill_pool_priority: chk_osd1_1=1 2020-03-11T22:45:42.188 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:375: TEST_backfill_pool_priority: chk_osd1_2=0 2020-03-11T22:45:42.188 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:364: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-03-11T22:45:42.188 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:366: TEST_backfill_pool_priority: ceph pg map 2.0 --format=json 2020-03-11T22:45:42.189 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:366: TEST_backfill_pool_priority: jq '.acting[]' 2020-03-11T22:45:42.587 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:367: TEST_backfill_pool_priority: head -1 td/osd-backfill-prio/acting 2020-03-11T22:45:42.588 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:367: TEST_backfill_pool_priority: local test_osd1=0 2020-03-11T22:45:42.589 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:368: TEST_backfill_pool_priority: tail -1 td/osd-backfill-prio/acting 2020-03-11T22:45:42.590 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:368: TEST_backfill_pool_priority: local test_osd2=1 2020-03-11T22:45:42.590 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:369: TEST_backfill_pool_priority: '[' -z 1.0 ']' 2020-03-11T22:45:42.590 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:376: TEST_backfill_pool_priority: '[' 1 '!=' 0 ']' 2020-03-11T22:45:42.590 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:378: TEST_backfill_pool_priority: PG2=2.0 2020-03-11T22:45:42.591 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:379: TEST_backfill_pool_priority: POOLNUM2=2 2020-03-11T22:45:42.591 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:380: TEST_backfill_pool_priority: pool2=test2 2020-03-11T22:45:42.591 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:381: TEST_backfill_pool_priority: chk_osd2_1=0 2020-03-11T22:45:42.591 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:382: TEST_backfill_pool_priority: chk_osd2_2=1 2020-03-11T22:45:42.591 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:383: TEST_backfill_pool_priority: break 2020-03-11T22:45:42.591 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:386: TEST_backfill_pool_priority: rm -f td/osd-backfill-prio/acting 2020-03-11T22:45:42.602 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:388: TEST_backfill_pool_priority: '[' test2 = '' ']' 2020-03-11T22:45:42.603 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:394: TEST_backfill_pool_priority: seq 1 3 2020-03-11T22:45:42.604 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:394: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-03-11T22:45:42.605 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:396: TEST_backfill_pool_priority: '[' 1 '!=' 1 -a 1 '!=' 2 ']' 2020-03-11T22:45:42.605 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:394: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-03-11T22:45:42.605 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:396: TEST_backfill_pool_priority: '[' 2 '!=' 1 -a 2 '!=' 2 ']' 2020-03-11T22:45:42.605 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:394: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-03-11T22:45:42.605 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:396: TEST_backfill_pool_priority: '[' 3 '!=' 1 -a 3 '!=' 2 ']' 2020-03-11T22:45:42.605 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:398: TEST_backfill_pool_priority: delete_pool test3 2020-03-11T22:45:42.606 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test3 2020-03-11T22:45:42.606 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test3 test3 --yes-i-really-really-mean-it 2020-03-11T22:45:43.261 INFO:tasks.workunit.client.0.smithi059.stderr:pool 'test3' does not exist 2020-03-11T22:45:43.269 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:402: TEST_backfill_pool_priority: pool1_extra_prio=1 2020-03-11T22:45:43.269 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:403: TEST_backfill_pool_priority: pool2_extra_prio=2 2020-03-11T22:45:43.270 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:405: TEST_backfill_pool_priority: expr 150 + 1 + 1 2020-03-11T22:45:43.271 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:405: TEST_backfill_pool_priority: pool1_prio=152 2020-03-11T22:45:43.272 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:406: TEST_backfill_pool_priority: expr 150 + 1 + 2 2020-03-11T22:45:43.272 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:406: TEST_backfill_pool_priority: pool2_prio=153 2020-03-11T22:45:43.272 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:408: TEST_backfill_pool_priority: ceph osd pool set test1 size 1 2020-03-11T22:45:43.907 INFO:tasks.workunit.client.0.smithi059.stderr:set pool 1 size to 1 2020-03-11T22:45:43.915 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:409: TEST_backfill_pool_priority: ceph osd pool set test1 recovery_priority 1 2020-03-11T22:45:44.692 INFO:tasks.workunit.client.0.smithi059.stderr:set pool 1 recovery_priority to 1 2020-03-11T22:45:44.700 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:410: TEST_backfill_pool_priority: ceph osd pool set test2 size 1 2020-03-11T22:45:45.570 INFO:tasks.workunit.client.0.smithi059.stderr:set pool 2 size to 1 2020-03-11T22:45:45.578 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:411: TEST_backfill_pool_priority: ceph osd pool set test2 recovery_priority 2 2020-03-11T22:45:46.196 INFO:tasks.workunit.client.0.smithi059.stderr:set pool 2 recovery_priority to 2 2020-03-11T22:45:46.204 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:412: TEST_backfill_pool_priority: wait_for_clean 2020-03-11T22:45:46.204 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:45:46.204 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:45:46.204 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:45:46.205 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:45:46.205 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:45:46.206 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:45:46.206 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:45:46.206 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:45:46.206 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:45:46.206 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:45:46.299 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:45:46.299 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:45:46.299 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:45:46.299 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:45:46.300 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:45:46.683 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:45:46.683 INFO:tasks.workunit.client.0.smithi059.stderr:1' 2020-03-11T22:45:46.683 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:45:46.683 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:45:46.684 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:45:46.829 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836490 2020-03-11T22:45:46.829 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836490' 2020-03-11T22:45:46.829 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:45:46.830 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:45:46.962 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672969 2020-03-11T22:45:46.962 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836490 1-42949672969' 2020-03-11T22:45:46.962 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:45:46.963 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836490 2020-03-11T22:45:46.963 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:45:46.964 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:45:46.965 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836490 2020-03-11T22:45:46.965 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:45:46.966 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836490 2020-03-11T22:45:46.966 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2020-03-11T22:45:46.966 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.0 seq 21474836490 2020-03-11T22:45:46.967 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:45:47.335 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836488 -lt 21474836490 2020-03-11T22:45:47.336 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:45:48.336 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:45:48.337 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:45:48.721 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836491 -lt 21474836490 2020-03-11T22:45:48.721 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:45:48.722 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672969 2020-03-11T22:45:48.722 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:45:48.722 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:45:48.724 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672969 2020-03-11T22:45:48.724 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:45:48.725 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672969 2020-03-11T22:45:48.725 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2020-03-11T22:45:48.725 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.1 seq 42949672969 2020-03-11T22:45:48.725 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:45:49.092 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672969 -lt 42949672969 2020-03-11T22:45:49.092 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:45:49.093 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:45:49.093 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:45:49.580 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 2 == 0 2020-03-11T22:45:49.580 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:45:49.581 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:45:49.581 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:45:49.581 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:45:49.581 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:45:49.582 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:45:49.582 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:45:49.933 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:45:49.934 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:45:49.934 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:45:49.935 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:45:50.411 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 2 2020-03-11T22:45:50.411 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:45:50.412 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:45:50.412 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:414: TEST_backfill_pool_priority: dd if=/dev/urandom of=td/osd-backfill-prio/data bs=1M count=10 2020-03-11T22:45:50.458 INFO:tasks.workunit.client.0.smithi059.stderr:10+0 records in 2020-03-11T22:45:50.458 INFO:tasks.workunit.client.0.smithi059.stderr:10+0 records out 2020-03-11T22:45:50.458 INFO:tasks.workunit.client.0.smithi059.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0459752 s, 228 MB/s 2020-03-11T22:45:50.458 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:415: TEST_backfill_pool_priority: p=1 2020-03-11T22:45:50.458 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:416: TEST_backfill_pool_priority: for pname in $pool1 $pool2 2020-03-11T22:45:50.459 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: seq 1 50 2020-03-11T22:45:50.459 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:45:50.459 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj1-p1 td/osd-backfill-prio/data 2020-03-11T22:45:50.775 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:45:50.775 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj2-p1 td/osd-backfill-prio/data 2020-03-11T22:45:50.984 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:45:50.984 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj3-p1 td/osd-backfill-prio/data 2020-03-11T22:45:51.339 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:45:51.339 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj4-p1 td/osd-backfill-prio/data 2020-03-11T22:45:51.559 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:45:51.559 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj5-p1 td/osd-backfill-prio/data 2020-03-11T22:45:51.809 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:45:51.810 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj6-p1 td/osd-backfill-prio/data 2020-03-11T22:45:52.051 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:45:52.051 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj7-p1 td/osd-backfill-prio/data 2020-03-11T22:45:52.335 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:45:52.590 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj8-p1 td/osd-backfill-prio/data 2020-03-11T22:45:52.602 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:45:52.602 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj9-p1 td/osd-backfill-prio/data 2020-03-11T22:45:52.871 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:45:52.871 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj10-p1 td/osd-backfill-prio/data 2020-03-11T22:45:53.127 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:45:53.127 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj11-p1 td/osd-backfill-prio/data 2020-03-11T22:45:53.414 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:45:53.414 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj12-p1 td/osd-backfill-prio/data 2020-03-11T22:45:53.668 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:45:53.669 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj13-p1 td/osd-backfill-prio/data 2020-03-11T22:45:53.919 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:45:53.919 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj14-p1 td/osd-backfill-prio/data 2020-03-11T22:45:54.196 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:45:54.196 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj15-p1 td/osd-backfill-prio/data 2020-03-11T22:45:54.427 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:45:54.427 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj16-p1 td/osd-backfill-prio/data 2020-03-11T22:45:54.802 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:45:54.802 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj17-p1 td/osd-backfill-prio/data 2020-03-11T22:45:55.061 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:45:55.061 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj18-p1 td/osd-backfill-prio/data 2020-03-11T22:45:55.369 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:45:55.369 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj19-p1 td/osd-backfill-prio/data 2020-03-11T22:45:55.619 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:45:55.620 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj20-p1 td/osd-backfill-prio/data 2020-03-11T22:45:55.893 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:45:55.894 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj21-p1 td/osd-backfill-prio/data 2020-03-11T22:45:56.119 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:45:56.120 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj22-p1 td/osd-backfill-prio/data 2020-03-11T22:45:56.387 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:45:56.387 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj23-p1 td/osd-backfill-prio/data 2020-03-11T22:45:56.678 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:45:56.678 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj24-p1 td/osd-backfill-prio/data 2020-03-11T22:45:56.939 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:45:56.939 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj25-p1 td/osd-backfill-prio/data 2020-03-11T22:45:57.178 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:45:57.179 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj26-p1 td/osd-backfill-prio/data 2020-03-11T22:45:57.403 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:45:57.404 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj27-p1 td/osd-backfill-prio/data 2020-03-11T22:45:57.712 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:45:57.712 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj28-p1 td/osd-backfill-prio/data 2020-03-11T22:45:57.971 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:45:57.971 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj29-p1 td/osd-backfill-prio/data 2020-03-11T22:45:58.254 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:45:58.254 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj30-p1 td/osd-backfill-prio/data 2020-03-11T22:45:58.550 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:45:58.551 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj31-p1 td/osd-backfill-prio/data 2020-03-11T22:45:58.821 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:45:58.821 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj32-p1 td/osd-backfill-prio/data 2020-03-11T22:45:59.079 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:45:59.080 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj33-p1 td/osd-backfill-prio/data 2020-03-11T22:45:59.369 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:45:59.369 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj34-p1 td/osd-backfill-prio/data 2020-03-11T22:45:59.630 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:45:59.630 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj35-p1 td/osd-backfill-prio/data 2020-03-11T22:45:59.920 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:45:59.920 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj36-p1 td/osd-backfill-prio/data 2020-03-11T22:46:00.186 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:00.186 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj37-p1 td/osd-backfill-prio/data 2020-03-11T22:46:00.490 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:00.490 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj38-p1 td/osd-backfill-prio/data 2020-03-11T22:46:00.705 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:00.706 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj39-p1 td/osd-backfill-prio/data 2020-03-11T22:46:00.973 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:00.973 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj40-p1 td/osd-backfill-prio/data 2020-03-11T22:46:01.247 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:01.247 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj41-p1 td/osd-backfill-prio/data 2020-03-11T22:46:01.524 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:01.524 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj42-p1 td/osd-backfill-prio/data 2020-03-11T22:46:01.756 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:01.756 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj43-p1 td/osd-backfill-prio/data 2020-03-11T22:46:01.997 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:01.998 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj44-p1 td/osd-backfill-prio/data 2020-03-11T22:46:02.273 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:02.273 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj45-p1 td/osd-backfill-prio/data 2020-03-11T22:46:02.560 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:02.560 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj46-p1 td/osd-backfill-prio/data 2020-03-11T22:46:02.848 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:02.848 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj47-p1 td/osd-backfill-prio/data 2020-03-11T22:46:03.106 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:03.107 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj48-p1 td/osd-backfill-prio/data 2020-03-11T22:46:03.398 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:03.398 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj49-p1 td/osd-backfill-prio/data 2020-03-11T22:46:03.665 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:03.665 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj50-p1 td/osd-backfill-prio/data 2020-03-11T22:46:03.947 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:422: TEST_backfill_pool_priority: expr 1 + 1 2020-03-11T22:46:03.948 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:422: TEST_backfill_pool_priority: p=2 2020-03-11T22:46:03.948 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:416: TEST_backfill_pool_priority: for pname in $pool1 $pool2 2020-03-11T22:46:03.949 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: seq 1 50 2020-03-11T22:46:03.950 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:03.950 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj1-p2 td/osd-backfill-prio/data 2020-03-11T22:46:04.247 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:04.247 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj2-p2 td/osd-backfill-prio/data 2020-03-11T22:46:04.531 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:04.531 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj3-p2 td/osd-backfill-prio/data 2020-03-11T22:46:04.831 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:04.831 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj4-p2 td/osd-backfill-prio/data 2020-03-11T22:46:05.085 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:05.085 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj5-p2 td/osd-backfill-prio/data 2020-03-11T22:46:05.433 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:05.433 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj6-p2 td/osd-backfill-prio/data 2020-03-11T22:46:05.677 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:05.677 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj7-p2 td/osd-backfill-prio/data 2020-03-11T22:46:05.926 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:05.927 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj8-p2 td/osd-backfill-prio/data 2020-03-11T22:46:06.227 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:06.227 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj9-p2 td/osd-backfill-prio/data 2020-03-11T22:46:06.452 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:06.452 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj10-p2 td/osd-backfill-prio/data 2020-03-11T22:46:06.802 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:06.802 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj11-p2 td/osd-backfill-prio/data 2020-03-11T22:46:07.010 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:07.010 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj12-p2 td/osd-backfill-prio/data 2020-03-11T22:46:07.285 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:07.286 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj13-p2 td/osd-backfill-prio/data 2020-03-11T22:46:07.519 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:07.519 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj14-p2 td/osd-backfill-prio/data 2020-03-11T22:46:07.813 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:07.813 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj15-p2 td/osd-backfill-prio/data 2020-03-11T22:46:08.061 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:08.061 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj16-p2 td/osd-backfill-prio/data 2020-03-11T22:46:08.347 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:08.348 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj17-p2 td/osd-backfill-prio/data 2020-03-11T22:46:08.695 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:08.695 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj18-p2 td/osd-backfill-prio/data 2020-03-11T22:46:08.945 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:08.945 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj19-p2 td/osd-backfill-prio/data 2020-03-11T22:46:09.236 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:09.237 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj20-p2 td/osd-backfill-prio/data 2020-03-11T22:46:09.495 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:09.495 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj21-p2 td/osd-backfill-prio/data 2020-03-11T22:46:09.786 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:09.787 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj22-p2 td/osd-backfill-prio/data 2020-03-11T22:46:10.053 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:10.054 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj23-p2 td/osd-backfill-prio/data 2020-03-11T22:46:10.377 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:10.377 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj24-p2 td/osd-backfill-prio/data 2020-03-11T22:46:10.679 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:10.679 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj25-p2 td/osd-backfill-prio/data 2020-03-11T22:46:10.961 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:10.962 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj26-p2 td/osd-backfill-prio/data 2020-03-11T22:46:11.262 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:11.262 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj27-p2 td/osd-backfill-prio/data 2020-03-11T22:46:11.579 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:11.579 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj28-p2 td/osd-backfill-prio/data 2020-03-11T22:46:11.829 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:11.829 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj29-p2 td/osd-backfill-prio/data 2020-03-11T22:46:12.114 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:12.114 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj30-p2 td/osd-backfill-prio/data 2020-03-11T22:46:12.363 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:12.363 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj31-p2 td/osd-backfill-prio/data 2020-03-11T22:46:12.646 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:12.647 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj32-p2 td/osd-backfill-prio/data 2020-03-11T22:46:12.946 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:12.946 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj33-p2 td/osd-backfill-prio/data 2020-03-11T22:46:13.163 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:13.163 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj34-p2 td/osd-backfill-prio/data 2020-03-11T22:46:13.430 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:13.430 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj35-p2 td/osd-backfill-prio/data 2020-03-11T22:46:13.888 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:13.889 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj36-p2 td/osd-backfill-prio/data 2020-03-11T22:46:14.155 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:14.155 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj37-p2 td/osd-backfill-prio/data 2020-03-11T22:46:14.397 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:14.397 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj38-p2 td/osd-backfill-prio/data 2020-03-11T22:46:14.730 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:14.730 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj39-p2 td/osd-backfill-prio/data 2020-03-11T22:46:14.994 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:14.994 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj40-p2 td/osd-backfill-prio/data 2020-03-11T22:46:15.281 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:15.281 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj41-p2 td/osd-backfill-prio/data 2020-03-11T22:46:15.514 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:15.514 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj42-p2 td/osd-backfill-prio/data 2020-03-11T22:46:15.789 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:15.790 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj43-p2 td/osd-backfill-prio/data 2020-03-11T22:46:16.006 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:16.006 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj44-p2 td/osd-backfill-prio/data 2020-03-11T22:46:16.297 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:16.298 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj45-p2 td/osd-backfill-prio/data 2020-03-11T22:46:16.590 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:16.590 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj46-p2 td/osd-backfill-prio/data 2020-03-11T22:46:16.891 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:16.891 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj47-p2 td/osd-backfill-prio/data 2020-03-11T22:46:17.140 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:17.140 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj48-p2 td/osd-backfill-prio/data 2020-03-11T22:46:17.425 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:17.425 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj49-p2 td/osd-backfill-prio/data 2020-03-11T22:46:17.665 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:46:17.665 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj50-p2 td/osd-backfill-prio/data 2020-03-11T22:46:17.917 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:422: TEST_backfill_pool_priority: expr 2 + 1 2020-03-11T22:46:17.917 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:422: TEST_backfill_pool_priority: p=3 2020-03-11T22:46:17.919 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:425: TEST_backfill_pool_priority: get_not_primary test1 obj1-p1 2020-03-11T22:46:17.919 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1166: get_not_primary: local poolname=test1 2020-03-11T22:46:17.919 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1167: get_not_primary: local objectname=obj1-p1 2020-03-11T22:46:17.920 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1169: get_not_primary: get_primary test1 obj1-p1 2020-03-11T22:46:17.920 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1133: get_primary: local poolname=test1 2020-03-11T22:46:17.920 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local objectname=obj1-p1 2020-03-11T22:46:17.920 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1136: get_primary: ceph --format json osd map test1 obj1-p1 2020-03-11T22:46:17.920 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: jq .acting_primary 2020-03-11T22:46:18.300 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1169: get_not_primary: local primary=1 2020-03-11T22:46:18.300 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: ceph --format json osd map test1 obj1-p1 2020-03-11T22:46:18.300 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2020-03-11T22:46:18.669 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:425: TEST_backfill_pool_priority: local otherosd=null 2020-03-11T22:46:18.670 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:427: TEST_backfill_pool_priority: ceph pg dump pgs 2020-03-11T22:46:19.013 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:46:19.013 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2020-03-11T22:45:43.946943+0000 38'150 38:192 [1] 1 [1] 1 0'0 2020-03-11T22:45:24.081865+0000 0'0 2020-03-11T22:45:24.081865+0000 0 2020-03-11T22:46:19.014 INFO:tasks.workunit.client.0.smithi059.stdout:2.0 49 0 0 0 0 513802240 0 0 47 47 active+clean 2020-03-11T22:45:45.461652+0000 38'147 38:186 [0] 0 [0] 0 0'0 2020-03-11T22:45:26.454442+0000 0'0 2020-03-11T22:45:26.454442+0000 0 2020-03-11T22:46:19.014 INFO:tasks.workunit.client.0.smithi059.stdout: 2020-03-11T22:46:19.014 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:46:19.015 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2020-03-11T22:46:19.023 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:428: TEST_backfill_pool_priority: ERRORS=0 2020-03-11T22:46:19.023 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:430: TEST_backfill_pool_priority: ceph osd pool set test1 size 2 2020-03-11T22:46:19.753 INFO:tasks.workunit.client.0.smithi059.stderr:set pool 1 size to 2 2020-03-11T22:46:19.761 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:431: TEST_backfill_pool_priority: ceph osd pool set test2 size 2 2020-03-11T22:46:20.815 INFO:tasks.workunit.client.0.smithi059.stderr:set pool 2 size to 2 2020-03-11T22:46:20.825 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:432: TEST_backfill_pool_priority: sleep 5 2020-03-11T22:46:25.828 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:433: TEST_backfill_pool_priority: get_asok_path osd.1 2020-03-11T22:46:25.828 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:46:25.828 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:46:25.828 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:46:25.828 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:25.828 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56462 2020-03-11T22:46:25.829 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56462/ceph-osd.1.asok 2020-03-11T22:46:25.829 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:433: TEST_backfill_pool_priority: CEPH_ARGS= 2020-03-11T22:46:25.829 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:433: TEST_backfill_pool_priority: ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:46:25.939 INFO:tasks.workunit.client.0.smithi059.stdout:osd.1 2020-03-11T22:46:25.940 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:434: TEST_backfill_pool_priority: echo osd.1 2020-03-11T22:46:25.940 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:435: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.1.out 2020-03-11T22:46:25.940 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2020-03-11T22:46:25.940 INFO:tasks.workunit.client.0.smithi059.stdout: "local_reservations": { 2020-03-11T22:46:25.940 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:46:25.941 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:46:25.941 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [], 2020-03-11T22:46:25.941 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [ 2020-03-11T22:46:25.941 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:46:25.941 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "1.0", 2020-03-11T22:46:25.941 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 152, 2020-03-11T22:46:25.941 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:46:25.942 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:46:25.942 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:46:25.942 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:46:25.942 INFO:tasks.workunit.client.0.smithi059.stdout: "remote_reservations": { 2020-03-11T22:46:25.942 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:46:25.942 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:46:25.943 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [], 2020-03-11T22:46:25.943 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [ 2020-03-11T22:46:25.943 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:46:25.943 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "2.0", 2020-03-11T22:46:25.943 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 153, 2020-03-11T22:46:25.943 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:46:25.943 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:46:25.943 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:46:25.944 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:46:25.944 INFO:tasks.workunit.client.0.smithi059.stdout:} 2020-03-11T22:46:25.945 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:436: TEST_backfill_pool_priority: get_asok_path osd.0 2020-03-11T22:46:25.945 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-11T22:46:25.945 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-11T22:46:25.945 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:46:25.946 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:25.946 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56462 2020-03-11T22:46:25.946 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56462/ceph-osd.0.asok 2020-03-11T22:46:25.946 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:436: TEST_backfill_pool_priority: CEPH_ARGS= 2020-03-11T22:46:25.946 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:436: TEST_backfill_pool_priority: ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.0.asok dump_recovery_reservations 2020-03-11T22:46:26.095 INFO:tasks.workunit.client.0.smithi059.stdout:osd.0 2020-03-11T22:46:26.095 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:437: TEST_backfill_pool_priority: echo osd.0 2020-03-11T22:46:26.095 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:438: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.0.out 2020-03-11T22:46:26.096 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2020-03-11T22:46:26.096 INFO:tasks.workunit.client.0.smithi059.stdout: "local_reservations": { 2020-03-11T22:46:26.096 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:46:26.096 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:46:26.096 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [], 2020-03-11T22:46:26.096 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [ 2020-03-11T22:46:26.097 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:46:26.097 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "2.0", 2020-03-11T22:46:26.097 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 153, 2020-03-11T22:46:26.097 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:46:26.097 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:46:26.097 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:46:26.097 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:46:26.098 INFO:tasks.workunit.client.0.smithi059.stdout: "remote_reservations": { 2020-03-11T22:46:26.098 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:46:26.098 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:46:26.098 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [], 2020-03-11T22:46:26.098 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [ 2020-03-11T22:46:26.098 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:46:26.098 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "1.0", 2020-03-11T22:46:26.099 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 152, 2020-03-11T22:46:26.099 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:46:26.099 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:46:26.099 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:46:26.099 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:46:26.099 INFO:tasks.workunit.client.0.smithi059.stdout:} 2020-03-11T22:46:26.100 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:441: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.1.out 2020-03-11T22:46:26.100 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:441: TEST_backfill_pool_priority: jq '.local_reservations.in_progress[0].item' 2020-03-11T22:46:26.100 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:441: TEST_backfill_pool_priority: eval 'ITEM="1.0"' 2020-03-11T22:46:26.101 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:441: TEST_backfill_pool_priority: ITEM=1.0 2020-03-11T22:46:26.101 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:442: TEST_backfill_pool_priority: '[' 1.0 '!=' 1.0 ']' 2020-03-11T22:46:26.101 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:447: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.1.out 2020-03-11T22:46:26.101 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:447: TEST_backfill_pool_priority: jq '.local_reservations.in_progress[0].prio' 2020-03-11T22:46:26.102 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:447: TEST_backfill_pool_priority: PRIO=152 2020-03-11T22:46:26.102 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:448: TEST_backfill_pool_priority: '[' 152 '!=' 152 ']' 2020-03-11T22:46:26.103 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:456: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.0.out 2020-03-11T22:46:26.104 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:456: TEST_backfill_pool_priority: jq '.remote_reservations.in_progress[0].item' 2020-03-11T22:46:26.106 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:456: TEST_backfill_pool_priority: eval 'ITEM="1.0"' 2020-03-11T22:46:26.106 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:456: TEST_backfill_pool_priority: ITEM=1.0 2020-03-11T22:46:26.106 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:457: TEST_backfill_pool_priority: '[' 1.0 '!=' 1.0 ']' 2020-03-11T22:46:26.107 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:462: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.0.out 2020-03-11T22:46:26.107 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:462: TEST_backfill_pool_priority: jq '.remote_reservations.in_progress[0].prio' 2020-03-11T22:46:26.109 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:462: TEST_backfill_pool_priority: PRIO=152 2020-03-11T22:46:26.109 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:463: TEST_backfill_pool_priority: '[' 152 '!=' 152 ']' 2020-03-11T22:46:26.110 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:471: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.0.out 2020-03-11T22:46:26.110 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:471: TEST_backfill_pool_priority: jq '.local_reservations.in_progress[0].item' 2020-03-11T22:46:26.113 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:471: TEST_backfill_pool_priority: eval 'ITEM="2.0"' 2020-03-11T22:46:26.113 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:471: TEST_backfill_pool_priority: ITEM=2.0 2020-03-11T22:46:26.113 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:472: TEST_backfill_pool_priority: '[' 2.0 '!=' 2.0 ']' 2020-03-11T22:46:26.114 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:477: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.0.out 2020-03-11T22:46:26.114 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:477: TEST_backfill_pool_priority: jq '.local_reservations.in_progress[0].prio' 2020-03-11T22:46:26.116 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:477: TEST_backfill_pool_priority: PRIO=153 2020-03-11T22:46:26.116 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:478: TEST_backfill_pool_priority: '[' 153 '!=' 153 ']' 2020-03-11T22:46:26.117 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:486: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.1.out 2020-03-11T22:46:26.117 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:486: TEST_backfill_pool_priority: jq '.remote_reservations.in_progress[0].item' 2020-03-11T22:46:26.120 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:486: TEST_backfill_pool_priority: eval 'ITEM="2.0"' 2020-03-11T22:46:26.120 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:486: TEST_backfill_pool_priority: ITEM=2.0 2020-03-11T22:46:26.120 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:487: TEST_backfill_pool_priority: '[' 2.0 '!=' 2.0 ']' 2020-03-11T22:46:26.121 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:492: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.1.out 2020-03-11T22:46:26.121 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:492: TEST_backfill_pool_priority: jq '.remote_reservations.in_progress[0].prio' 2020-03-11T22:46:26.123 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:492: TEST_backfill_pool_priority: PRIO=153 2020-03-11T22:46:26.124 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:493: TEST_backfill_pool_priority: '[' 153 '!=' 153 ']' 2020-03-11T22:46:26.124 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:500: TEST_backfill_pool_priority: wait_for_clean 2020-03-11T22:46:26.124 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:46:26.124 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:46:26.124 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:46:26.125 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:46:26.126 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:46:26.126 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:46:26.126 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:46:26.126 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:46:26.126 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:46:26.127 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:46:26.217 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:46:26.218 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:46:26.218 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:46:26.218 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:46:26.218 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:46:26.808 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:46:26.808 INFO:tasks.workunit.client.0.smithi059.stderr:1' 2020-03-11T22:46:26.808 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:46:26.808 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:46:26.809 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:46:27.041 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836500 2020-03-11T22:46:27.041 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836500' 2020-03-11T22:46:27.041 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:46:27.042 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:46:27.174 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672979 2020-03-11T22:46:27.174 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836500 1-42949672979' 2020-03-11T22:46:27.174 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:46:27.175 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836500 2020-03-11T22:46:27.175 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:46:27.176 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:46:27.177 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836500 2020-03-11T22:46:27.177 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:46:27.178 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836500 2020-03-11T22:46:27.178 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2020-03-11T22:46:27.178 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.0 seq 21474836500 2020-03-11T22:46:27.178 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:46:27.614 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836498 -lt 21474836500 2020-03-11T22:46:27.614 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:46:28.615 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:46:28.616 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:46:29.160 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836501 -lt 21474836500 2020-03-11T22:46:29.161 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:46:29.161 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672979 2020-03-11T22:46:29.162 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:46:29.162 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:46:29.163 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672979 2020-03-11T22:46:29.164 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:46:29.164 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672979 2020-03-11T22:46:29.165 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2020-03-11T22:46:29.165 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.1 seq 42949672979 2020-03-11T22:46:29.165 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:46:29.534 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672979 -lt 42949672979 2020-03-11T22:46:29.536 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:46:29.536 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:46:29.536 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:46:30.056 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 2 == 0 2020-03-11T22:46:30.057 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:46:30.058 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:46:30.058 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:46:30.058 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:46:30.058 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:46:30.058 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:46:30.059 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:46:30.414 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:46:30.415 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:46:30.415 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:46:30.416 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:46:30.992 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:46:30.993 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:46:30.993 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:46:30.993 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:46:30.993 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:46:30.993 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:46:31.093 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:46:31.093 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:46:31.094 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:46:31.095 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:46:31.095 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:46:31.095 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:46:31.095 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:46:31.095 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:46:31.452 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:46:31.453 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:46:31.453 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:46:31.453 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:46:31.939 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:46:31.939 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:46:31.939 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:46:31.939 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:46:31.939 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:46:31.940 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:46:31.940 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:46:31.940 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:46:31.940 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:46:32.423 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=28373714 2020-03-11T22:46:32.423 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 28373714 '!=' null 2020-03-11T22:46:32.423 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:46:32.423 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:46:32.423 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:46:32.524 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:46:32.524 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:46:32.525 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:46:32.525 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:46:32.525 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:46:32.525 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:46:32.526 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:46:32.526 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:46:32.901 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:46:32.902 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:46:32.902 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:46:32.902 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:46:33.375 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:46:33.375 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:46:33.375 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:46:33.375 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:46:33.375 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:46:33.375 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:46:33.375 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:46:33.376 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:46:33.376 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:46:33.858 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=24217844 2020-03-11T22:46:33.858 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 24217844 '!=' null 2020-03-11T22:46:33.858 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:46:33.858 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:46:33.858 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:46:33.958 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:46:33.959 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:46:33.960 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:46:33.960 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:46:33.960 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:46:33.960 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:46:33.960 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:46:33.961 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:46:34.321 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:46:34.321 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:46:34.322 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:46:34.322 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:46:34.817 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:46:34.818 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:46:34.818 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:46:34.818 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:46:34.818 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:46:34.818 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:46:34.818 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:46:34.819 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:46:34.819 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:46:35.436 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=36693437 2020-03-11T22:46:35.437 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 36693437 '!=' null 2020-03-11T22:46:35.437 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:46:35.437 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:46:35.437 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:46:35.537 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:46:35.538 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:46:35.539 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:46:35.539 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:46:35.539 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:46:35.539 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:46:35.539 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:46:35.540 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:46:35.897 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:46:35.898 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:46:35.898 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:46:35.898 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:46:36.376 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:46:36.376 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:46:36.376 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:46:36.376 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:46:36.376 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:46:36.376 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:46:36.377 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:46:36.377 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:46:36.377 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:46:37.037 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=29704744 2020-03-11T22:46:37.037 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 29704744 '!=' null 2020-03-11T22:46:37.037 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:46:37.037 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:46:37.037 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:46:37.138 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:46:37.138 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:46:37.139 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:46:37.139 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:46:37.139 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:46:37.139 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:46:37.139 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:46:37.140 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:46:37.496 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:46:37.497 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:46:37.497 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:46:37.497 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:46:37.981 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:46:37.982 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:46:37.982 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:46:37.982 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:46:37.982 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:46:37.982 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:46:37.982 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:46:37.983 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:46:37.983 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:46:38.592 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=29704744 2020-03-11T22:46:38.592 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 29704744 '!=' null 2020-03-11T22:46:38.592 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:46:38.592 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:46:38.592 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:46:38.692 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:46:38.693 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:46:38.694 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:46:38.694 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:46:38.694 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:46:38.694 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:46:38.694 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:46:38.695 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:46:39.055 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:46:39.056 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:46:39.056 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:46:39.056 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:46:39.536 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:46:39.536 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:46:39.537 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:46:39.537 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:46:39.537 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:46:39.537 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:46:39.537 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:46:39.538 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:46:39.538 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:46:40.076 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=36693931 2020-03-11T22:46:40.076 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 36693931 '!=' null 2020-03-11T22:46:40.076 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:46:40.076 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:46:40.076 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:46:40.176 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:46:40.177 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:46:40.178 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:46:40.178 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:46:40.178 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:46:40.178 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:46:40.178 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:46:40.178 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:46:40.644 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:46:40.645 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:46:40.645 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:46:40.645 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:46:41.154 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:46:41.155 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:46:41.155 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:46:41.155 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:46:41.155 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:46:41.155 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:46:41.155 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:46:41.156 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:46:41.156 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:46:41.705 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=30578252 2020-03-11T22:46:41.706 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 30578252 '!=' null 2020-03-11T22:46:41.706 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:46:41.706 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:46:41.706 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:46:41.806 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:46:41.806 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:46:41.807 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:46:41.808 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:46:41.808 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:46:41.808 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:46:41.808 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:46:41.808 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:46:42.180 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:46:42.181 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:46:42.181 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:46:42.182 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:46:42.850 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:46:42.850 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:46:42.850 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:46:42.850 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:46:42.850 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:46:42.850 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:46:42.851 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:46:42.851 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:46:42.851 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:46:43.387 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=27957287 2020-03-11T22:46:43.388 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 27957287 '!=' null 2020-03-11T22:46:43.388 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:46:43.388 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:46:43.388 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:46:43.488 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:46:43.488 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:46:43.489 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:46:43.490 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:46:43.490 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:46:43.490 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:46:43.490 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:46:43.490 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:46:43.906 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:46:43.906 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:46:43.907 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:46:43.907 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:46:44.377 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:46:44.377 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:46:44.377 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:46:44.377 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:46:44.378 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:46:44.378 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:46:44.378 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:46:44.378 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:46:44.379 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:46:44.859 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=27957287 2020-03-11T22:46:44.859 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 27957287 '!=' null 2020-03-11T22:46:44.859 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:46:44.860 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:46:44.860 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:46:44.960 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:46:44.960 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:46:44.961 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:46:44.961 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:46:44.961 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:46:44.962 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:46:44.962 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:46:44.962 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:46:45.324 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:46:45.325 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:46:45.325 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:46:45.325 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:46:45.998 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:46:45.998 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:46:45.998 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:46:45.998 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:46:45.998 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:46:45.999 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:46:45.999 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:46:45.999 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:46:46.000 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:46:46.561 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=41936118 2020-03-11T22:46:46.561 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 41936118 '!=' null 2020-03-11T22:46:46.562 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:46:46.562 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:46:46.562 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:46:46.662 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:46:46.662 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:46:46.663 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:46:46.663 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:46:46.663 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:46:46.663 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:46:46.664 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:46:46.664 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:46:47.029 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:46:47.030 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:46:47.030 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:46:47.030 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:46:47.505 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:46:47.505 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:46:47.505 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:46:47.505 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:46:47.505 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:46:47.506 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:46:47.506 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:46:47.506 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:46:47.506 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:46:48.117 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=27957475 2020-03-11T22:46:48.117 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 27957475 '!=' null 2020-03-11T22:46:48.117 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:46:48.117 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:46:48.117 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:46:48.217 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:46:48.218 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:46:48.219 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:46:48.219 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:46:48.219 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:46:48.219 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:46:48.219 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:46:48.220 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:46:48.708 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:46:48.708 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:46:48.709 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:46:48.709 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:46:49.198 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:46:49.199 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:46:49.199 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:46:49.199 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:46:49.199 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:46:49.199 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:46:49.199 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:46:49.200 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:46:49.200 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:46:49.678 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=34946576 2020-03-11T22:46:49.678 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 34946576 '!=' null 2020-03-11T22:46:49.679 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:46:49.679 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:46:49.679 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:46:49.779 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:46:49.779 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:46:49.780 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:46:49.780 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:46:49.780 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:46:49.781 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:46:49.781 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:46:49.781 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:46:50.142 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:46:50.142 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:46:50.143 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:46:50.143 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:46:50.624 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:46:50.624 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:46:50.624 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:46:50.625 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:46:50.625 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:46:50.625 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:46:50.625 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:46:50.626 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:46:50.626 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:46:51.185 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=34946622 2020-03-11T22:46:51.186 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 34946622 '!=' null 2020-03-11T22:46:51.186 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:46:51.186 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:46:51.186 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:46:51.286 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:46:51.286 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:46:51.287 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:46:51.288 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:46:51.288 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:46:51.288 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:46:51.288 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:46:51.288 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:46:51.646 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:46:51.647 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:46:51.647 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:46:51.647 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:46:52.154 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:46:52.155 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:46:52.155 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:46:52.155 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:46:52.155 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:46:52.155 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:46:52.155 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:46:52.156 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:46:52.156 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:46:52.633 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=20967671 2020-03-11T22:46:52.634 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 20967671 '!=' null 2020-03-11T22:46:52.634 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:46:52.634 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:46:52.634 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:46:52.734 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:46:52.734 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:46:52.735 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:46:52.736 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:46:52.736 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:46:52.736 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:46:52.736 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:46:52.736 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:46:53.095 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:46:53.096 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:46:53.096 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:46:53.096 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:46:53.570 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:46:53.571 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:46:53.571 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:46:53.571 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:46:53.571 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:46:53.571 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:46:53.571 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:46:53.572 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:46:53.572 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:46:54.053 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=20967671 2020-03-11T22:46:54.054 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 20967671 '!=' null 2020-03-11T22:46:54.054 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:46:54.054 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:46:54.054 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:46:54.154 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:46:54.154 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:46:54.155 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:46:54.156 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:46:54.156 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:46:54.156 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:46:54.156 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:46:54.156 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:46:54.529 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:46:54.530 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:46:54.530 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:46:54.530 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:46:54.986 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 2 2020-03-11T22:46:54.986 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:46:54.986 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:46:54.987 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:502: TEST_backfill_pool_priority: '[' 0 '!=' 0 ']' 2020-03-11T22:46:54.987 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:506: TEST_backfill_pool_priority: echo TEST PASSED 2020-03-11T22:46:54.987 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:509: TEST_backfill_pool_priority: delete_pool test1 2020-03-11T22:46:54.987 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test1 2020-03-11T22:46:54.987 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test1 test1 --yes-i-really-really-mean-it 2020-03-11T22:46:54.987 INFO:tasks.workunit.client.0.smithi059.stdout:TEST PASSED 2020-03-11T22:46:55.492 INFO:tasks.workunit.client.0.smithi059.stderr:pool 'test1' does not exist 2020-03-11T22:46:55.500 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:510: TEST_backfill_pool_priority: delete_pool test2 2020-03-11T22:46:55.500 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test2 2020-03-11T22:46:55.500 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test2 test2 --yes-i-really-really-mean-it 2020-03-11T22:46:55.971 INFO:tasks.workunit.client.0.smithi059.stderr:pool 'test2' does not exist 2020-03-11T22:46:55.979 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:511: TEST_backfill_pool_priority: kill_daemons td/osd-backfill-prio 2020-03-11T22:46:55.981 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:46:55.981 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:46:55.981 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:46:55.981 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:46:55.981 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:46:56.096 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:46:56.096 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:512: TEST_backfill_pool_priority: return 0 2020-03-11T22:46:56.096 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:40: run: teardown td/osd-backfill-prio 2020-03-11T22:46:56.096 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-prio 2020-03-11T22:46:56.097 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:46:56.097 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-prio KILL 2020-03-11T22:46:56.097 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:46:56.097 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:46:56.097 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:46:56.097 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:46:56.097 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:46:56.100 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:46:56.101 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:46:56.101 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:46:56.101 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:46:56.102 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:46:56.102 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:46:56.103 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:46:56.104 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:46:56.105 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:46:56.105 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:46:56.105 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:46:56.105 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:46:56.106 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:46:56.106 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-prio 2020-03-11T22:46:56.118 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:46:56.118 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:56.118 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56462 2020-03-11T22:46:56.118 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.56462 2020-03-11T22:46:56.119 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:46:56.120 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:46:56.120 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:37: run: for func in $funcs 2020-03-11T22:46:56.120 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:38: run: setup td/osd-backfill-prio 2020-03-11T22:46:56.120 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-prio 2020-03-11T22:46:56.120 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-prio 2020-03-11T22:46:56.120 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-prio 2020-03-11T22:46:56.121 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:46:56.121 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-prio KILL 2020-03-11T22:46:56.121 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:46:56.121 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:46:56.121 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:46:56.121 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:46:56.121 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:46:56.122 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:46:56.122 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:46:56.122 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:46:56.123 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:46:56.123 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:46:56.123 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:46:56.124 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:46:56.125 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:46:56.125 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:46:56.125 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:46:56.126 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:46:56.126 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:46:56.126 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:46:56.127 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-prio 2020-03-11T22:46:56.128 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:46:56.128 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:56.128 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56462 2020-03-11T22:46:56.128 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.56462 2020-03-11T22:46:56.129 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:46:56.129 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:46:56.129 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-prio 2020-03-11T22:46:56.130 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-11T22:46:56.130 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:56.130 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56462 2020-03-11T22:46:56.130 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.56462 2020-03-11T22:46:56.132 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-11T22:46:56.132 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-03-11T22:46:56.132 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-11T22:46:56.132 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-prio 1' TERM HUP INT 2020-03-11T22:46:56.132 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:39: run: TEST_backfill_priority td/osd-backfill-prio 2020-03-11T22:46:56.132 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:46: TEST_backfill_priority: local dir=td/osd-backfill-prio 2020-03-11T22:46:56.133 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:47: TEST_backfill_priority: local pools=10 2020-03-11T22:46:56.133 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:48: TEST_backfill_priority: local OSDS=5 2020-03-11T22:46:56.133 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:50: TEST_backfill_priority: expr 150 + 1 2020-03-11T22:46:56.134 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:50: TEST_backfill_priority: local degraded_prio=151 2020-03-11T22:46:56.134 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:51: TEST_backfill_priority: local max_tries=10 2020-03-11T22:46:56.134 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:53: TEST_backfill_priority: run_mon td/osd-backfill-prio a 2020-03-11T22:46:56.134 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-prio 2020-03-11T22:46:56.135 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-11T22:46:56.135 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-11T22:46:56.135 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-11T22:46:56.135 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-prio/a 2020-03-11T22:46:56.135 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-prio/a --run-dir=td/osd-backfill-prio 2020-03-11T22:46:56.468 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-11T22:46:56.468 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:46:56.468 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:46:56.468 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:46:56.468 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:56.468 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56462 2020-03-11T22:46:56.469 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.56462/$cluster-$name.asok' 2020-03-11T22:46:56.469 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-prio/a '--log-file=td/osd-backfill-prio/$name.log' '--admin-socket=/tmp/ceph-asok.56462/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-prio/log --run-dir=td/osd-backfill-prio '--pid-file=td/osd-backfill-prio/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-11T22:46:56.619 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-11T22:46:56.620 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-11T22:46:56.621 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:46:56.621 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:46:56.621 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-11T22:46:56.621 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-11T22:46:56.621 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:46:56.621 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:46:56.622 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:46:56.622 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:46:56.622 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:56.622 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56462 2020-03-11T22:46:56.622 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56462/ceph-mon.a.asok 2020-03-11T22:46:56.623 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:46:56.623 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.56462/ceph-mon.a.asok config get fsid 2020-03-11T22:46:56.716 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-11T22:46:56.716 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:46:56.716 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:46:56.717 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-11T22:46:56.717 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-11T22:46:56.717 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:46:56.717 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:46:56.718 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:46:56.718 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:46:56.718 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:56.718 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56462 2020-03-11T22:46:56.719 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56462/ceph-mon.a.asok 2020-03-11T22:46:56.719 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:46:56.719 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.56462/ceph-mon.a.asok config get mon_host 2020-03-11T22:46:56.804 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:54: TEST_backfill_priority: run_mgr td/osd-backfill-prio x 2020-03-11T22:46:56.804 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/osd-backfill-prio 2020-03-11T22:46:56.804 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-11T22:46:56.804 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-11T22:46:56.804 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-11T22:46:56.805 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/osd-backfill-prio/x 2020-03-11T22:46:56.805 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-11T22:46:57.730 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-11T22:46:57.731 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:46:57.731 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:46:57.731 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:46:57.731 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:57.731 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56462 2020-03-11T22:46:57.731 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.56462/$cluster-$name.asok' 2020-03-11T22:46:57.732 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:46:57.732 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-prio/x '--log-file=td/osd-backfill-prio/$name.log' '--admin-socket=/tmp/ceph-asok.56462/$cluster-$name.asok' --run-dir=td/osd-backfill-prio '--pid-file=td/osd-backfill-prio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:46:57.755 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:55: TEST_backfill_priority: export CEPH_ARGS 2020-03-11T22:46:57.756 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: expr 5 - 1 2020-03-11T22:46:57.757 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: seq 0 4 2020-03-11T22:46:57.761 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:46:57.761 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:59: TEST_backfill_priority: run_osd td/osd-backfill-prio 0 2020-03-11T22:46:57.761 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-prio 2020-03-11T22:46:57.762 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:46:57.762 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-11T22:46:57.762 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:46:57.762 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-prio/0 2020-03-11T22:46:57.762 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=34f3141b-0483-4f39-a129-ea18fdfbfb6a --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-03-11T22:46:57.762 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:46:57.763 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:46:57.763 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:46:57.763 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/0' 2020-03-11T22:46:57.763 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/0/journal' 2020-03-11T22:46:57.763 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:46:57.763 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:46:57.763 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2020-03-11T22:46:57.764 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:46:57.764 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:46:57.764 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:46:57.764 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:46:57.764 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:57.764 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56462 2020-03-11T22:46:57.764 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.56462/$cluster-$name.asok' 2020-03-11T22:46:57.765 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56462/$cluster-$name.asok' 2020-03-11T22:46:57.765 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:46:57.765 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:46:57.765 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:46:57.765 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2020-03-11T22:46:57.765 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2020-03-11T22:46:57.765 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:46:57.766 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:46:57.766 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:46:57.766 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:46:57.766 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:46:57.766 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-prio/0 2020-03-11T22:46:57.766 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:46:57.766 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=daabb136-f3c1-4b2d-bc06-ad677db57ac9 2020-03-11T22:46:57.767 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 daabb136-f3c1-4b2d-bc06-ad677db57ac9' 2020-03-11T22:46:57.767 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:46:57.767 INFO:tasks.workunit.client.0.smithi059.stdout:add osd0 daabb136-f3c1-4b2d-bc06-ad677db57ac9 2020-03-11T22:46:57.778 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBhamle1Yk3LhAAG4eyCCRPGCYwM8PTWJ3trw== 2020-03-11T22:46:57.778 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBhamle1Yk3LhAAG4eyCCRPGCYwM8PTWJ3trw=="}' 2020-03-11T22:46:57.778 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new daabb136-f3c1-4b2d-bc06-ad677db57ac9 -i td/osd-backfill-prio/0/new.json 2020-03-11T22:46:58.113 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:46:58.122 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-prio/0/new.json 2020-03-11T22:46:58.125 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=34f3141b-0483-4f39-a129-ea18fdfbfb6a --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/0 --osd-journal=td/osd-backfill-prio/0/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56462/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBhamle1Yk3LhAAG4eyCCRPGCYwM8PTWJ3trw== --osd-uuid daabb136-f3c1-4b2d-bc06-ad677db57ac9 2020-03-11T22:46:58.150 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:46:58.147+0000 7ff4d5a44ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:58.151 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:46:58.149+0000 7ff4d5a44ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:58.153 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:46:58.151+0000 7ff4d5a44ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:58.154 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:46:58.151+0000 7ff4d5a44ec0 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2020-03-11T22:46:58.154 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:46:58.151+0000 7ff4d5a44ec0 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2020-03-11T22:46:58.154 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:46:58.151+0000 7ff4d5a44ec0 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2020-03-11T22:46:58.268 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:46:58.265+0000 7ff4d5a44ec0 -1 bluestore(td/osd-backfill-prio/0) _read_fsid unparsable uuid 2020-03-11T22:47:01.332 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-prio/0/keyring 2020-03-11T22:47:01.332 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:47:01.333 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-11T22:47:01.333 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-prio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:47:01.333 INFO:tasks.workunit.client.0.smithi059.stdout:adding osd0 key to auth repository 2020-03-11T22:47:01.835 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-11T22:47:01.835 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=34f3141b-0483-4f39-a129-ea18fdfbfb6a --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/0 --osd-journal=td/osd-backfill-prio/0/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56462/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:47:01.835 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:47:01.836 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:47:01.836 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:47:01.836 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.0 2020-03-11T22:47:01.859 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:47:01.857+0000 7fe608370ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:01.861 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:47:01.859+0000 7fe608370ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:01.863 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:47:01.860+0000 7fe608370ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:02.204 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-11T22:47:02.205 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:47:02.205 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:47:02.205 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:47:02.205 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:47:02.205 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:47:02.205 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:47:02.206 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:47:02.206 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:47:02.206 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:47:02.378 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:47:02.376+0000 7fe608370ec0 -1 Falling back to public interface 2020-03-11T22:47:02.572 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:47:03.133 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:47:03.131+0000 7fe608370ec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-11T22:47:03.574 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:47:03.574 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:47:03.574 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:47:03.574 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:47:03.574 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:47:03.575 INFO:tasks.workunit.client.0.smithi059.stdout:1 2020-03-11T22:47:03.968 INFO:tasks.workunit.client.0.smithi059.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/63184,v1:127.0.0.1:6803/63184] [v2:127.0.0.1:6804/63184,v1:127.0.0.1:6805/63184] exists,up daabb136-f3c1-4b2d-bc06-ad677db57ac9 2020-03-11T22:47:03.969 INFO:tasks.workunit.client.0.smithi059.stdout:add osd1 627f2e9f-b510-4d15-bbb8-f42ed6a4eb53 2020-03-11T22:47:03.969 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:47:03.969 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:47:03.969 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:47:03.969 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:47:03.970 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:59: TEST_backfill_priority: run_osd td/osd-backfill-prio 1 2020-03-11T22:47:03.970 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-prio 2020-03-11T22:47:03.970 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:47:03.970 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-11T22:47:03.970 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:47:03.970 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-prio/1 2020-03-11T22:47:03.970 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=34f3141b-0483-4f39-a129-ea18fdfbfb6a --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-03-11T22:47:03.971 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:47:03.971 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:47:03.971 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:47:03.971 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/1' 2020-03-11T22:47:03.971 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/1/journal' 2020-03-11T22:47:03.971 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:47:03.971 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:47:03.972 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2020-03-11T22:47:03.972 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:47:03.972 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:47:03.972 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:47:03.972 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:47:03.972 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:47:03.972 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56462 2020-03-11T22:47:03.972 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.56462/$cluster-$name.asok' 2020-03-11T22:47:03.973 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56462/$cluster-$name.asok' 2020-03-11T22:47:03.973 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:47:03.973 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:47:03.973 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:47:03.973 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2020-03-11T22:47:03.973 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2020-03-11T22:47:03.973 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:47:03.974 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:47:03.974 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:47:03.974 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:47:03.974 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:47:03.974 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-prio/1 2020-03-11T22:47:03.974 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:47:03.974 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=627f2e9f-b510-4d15-bbb8-f42ed6a4eb53 2020-03-11T22:47:03.975 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 627f2e9f-b510-4d15-bbb8-f42ed6a4eb53' 2020-03-11T22:47:03.975 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:47:03.985 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBnamleUtSNOhAADHNT8aszWD0hnOlMJgFa8g== 2020-03-11T22:47:03.985 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBnamleUtSNOhAADHNT8aszWD0hnOlMJgFa8g=="}' 2020-03-11T22:47:03.985 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 627f2e9f-b510-4d15-bbb8-f42ed6a4eb53 -i td/osd-backfill-prio/1/new.json 2020-03-11T22:47:04.473 INFO:tasks.workunit.client.0.smithi059.stdout:1 2020-03-11T22:47:04.483 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-prio/1/new.json 2020-03-11T22:47:04.484 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=34f3141b-0483-4f39-a129-ea18fdfbfb6a --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/1 --osd-journal=td/osd-backfill-prio/1/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56462/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBnamleUtSNOhAADHNT8aszWD0hnOlMJgFa8g== --osd-uuid 627f2e9f-b510-4d15-bbb8-f42ed6a4eb53 2020-03-11T22:47:04.511 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:47:04.508+0000 7fa26f807ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:04.512 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:47:04.510+0000 7fa26f807ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:04.514 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:47:04.511+0000 7fa26f807ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:04.515 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:47:04.512+0000 7fa26f807ec0 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2020-03-11T22:47:04.515 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:47:04.512+0000 7fa26f807ec0 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2020-03-11T22:47:04.515 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:47:04.512+0000 7fa26f807ec0 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2020-03-11T22:47:04.557 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:47:04.554+0000 7fa26f807ec0 -1 bluestore(td/osd-backfill-prio/1) _read_fsid unparsable uuid 2020-03-11T22:47:07.742 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-prio/1/keyring 2020-03-11T22:47:07.743 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:47:07.744 INFO:tasks.workunit.client.0.smithi059.stdout:adding osd1 key to auth repository 2020-03-11T22:47:07.744 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-03-11T22:47:07.744 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-prio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:47:08.238 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-03-11T22:47:08.238 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=34f3141b-0483-4f39-a129-ea18fdfbfb6a --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/1 --osd-journal=td/osd-backfill-prio/1/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56462/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:47:08.238 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:47:08.238 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:47:08.238 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:47:08.239 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.1 2020-03-11T22:47:08.263 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:47:08.260+0000 7f2dd1cf9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:08.265 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:47:08.262+0000 7f2dd1cf9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:08.267 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:47:08.264+0000 7f2dd1cf9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:08.603 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-03-11T22:47:08.603 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:47:08.604 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:47:08.604 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:47:08.604 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:47:08.604 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:47:08.604 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:47:08.604 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:47:08.604 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:47:08.605 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:47:08.786 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:47:08.783+0000 7f2dd1cf9ec0 -1 Falling back to public interface 2020-03-11T22:47:09.031 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:47:09.480 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:47:09.478+0000 7f2dd1cf9ec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-11T22:47:10.033 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:47:10.033 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:47:10.033 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:47:10.034 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:47:10.034 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:47:10.034 INFO:tasks.workunit.client.0.smithi059.stdout:1 2020-03-11T22:47:10.417 INFO:tasks.workunit.client.0.smithi059.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/63539,v1:127.0.0.1:6811/63539] [v2:127.0.0.1:6812/63539,v1:127.0.0.1:6813/63539] exists,up 627f2e9f-b510-4d15-bbb8-f42ed6a4eb53 2020-03-11T22:47:10.418 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:47:10.418 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:47:10.418 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:47:10.418 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:47:10.418 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:59: TEST_backfill_priority: run_osd td/osd-backfill-prio 2 2020-03-11T22:47:10.418 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-prio 2020-03-11T22:47:10.418 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:47:10.419 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-03-11T22:47:10.419 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:47:10.419 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-prio/2 2020-03-11T22:47:10.419 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=34f3141b-0483-4f39-a129-ea18fdfbfb6a --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-03-11T22:47:10.419 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:47:10.419 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:47:10.419 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:47:10.420 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/2' 2020-03-11T22:47:10.420 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/2/journal' 2020-03-11T22:47:10.420 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:47:10.420 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:47:10.420 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2020-03-11T22:47:10.420 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:47:10.420 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:47:10.421 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:47:10.421 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:47:10.421 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:47:10.421 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56462 2020-03-11T22:47:10.421 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.56462/$cluster-$name.asok' 2020-03-11T22:47:10.421 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56462/$cluster-$name.asok' 2020-03-11T22:47:10.421 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:47:10.421 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:47:10.422 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:47:10.422 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2020-03-11T22:47:10.422 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2020-03-11T22:47:10.422 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:47:10.422 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:47:10.422 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:47:10.422 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:47:10.423 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:47:10.423 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-prio/2 2020-03-11T22:47:10.423 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:47:10.423 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=f739d588-0568-4527-ab7b-85ef8fc4e62b 2020-03-11T22:47:10.424 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 f739d588-0568-4527-ab7b-85ef8fc4e62b' 2020-03-11T22:47:10.424 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:47:10.424 INFO:tasks.workunit.client.0.smithi059.stdout:add osd2 f739d588-0568-4527-ab7b-85ef8fc4e62b 2020-03-11T22:47:10.434 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBuamlek7i9GRAApEi4LyeCzdx3Zr5ryGRw7A== 2020-03-11T22:47:10.434 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBuamlek7i9GRAApEi4LyeCzdx3Zr5ryGRw7A=="}' 2020-03-11T22:47:10.434 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new f739d588-0568-4527-ab7b-85ef8fc4e62b -i td/osd-backfill-prio/2/new.json 2020-03-11T22:47:10.941 INFO:tasks.workunit.client.0.smithi059.stdout:2 2020-03-11T22:47:10.950 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-prio/2/new.json 2020-03-11T22:47:10.951 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=34f3141b-0483-4f39-a129-ea18fdfbfb6a --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/2 --osd-journal=td/osd-backfill-prio/2/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56462/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBuamlek7i9GRAApEi4LyeCzdx3Zr5ryGRw7A== --osd-uuid f739d588-0568-4527-ab7b-85ef8fc4e62b 2020-03-11T22:47:10.978 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:47:10.975+0000 7fd3ff0b8ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:11.005 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:47:11.002+0000 7fd3ff0b8ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:11.007 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:47:11.004+0000 7fd3ff0b8ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:11.007 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:47:11.005+0000 7fd3ff0b8ec0 -1 bluestore(td/osd-backfill-prio/2/block) _read_bdev_label failed to open td/osd-backfill-prio/2/block: (2) No such file or directory 2020-03-11T22:47:11.008 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:47:11.005+0000 7fd3ff0b8ec0 -1 bluestore(td/osd-backfill-prio/2/block) _read_bdev_label failed to open td/osd-backfill-prio/2/block: (2) No such file or directory 2020-03-11T22:47:11.008 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:47:11.005+0000 7fd3ff0b8ec0 -1 bluestore(td/osd-backfill-prio/2/block) _read_bdev_label failed to open td/osd-backfill-prio/2/block: (2) No such file or directory 2020-03-11T22:47:11.096 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:47:11.094+0000 7fd3ff0b8ec0 -1 bluestore(td/osd-backfill-prio/2) _read_fsid unparsable uuid 2020-03-11T22:47:14.297 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-prio/2/keyring 2020-03-11T22:47:14.298 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:47:14.299 INFO:tasks.workunit.client.0.smithi059.stdout:adding osd2 key to auth repository 2020-03-11T22:47:14.299 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-03-11T22:47:14.299 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-prio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:47:14.797 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-03-11T22:47:14.797 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:47:14.797 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:47:14.797 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:47:14.797 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=34f3141b-0483-4f39-a129-ea18fdfbfb6a --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/2 --osd-journal=td/osd-backfill-prio/2/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56462/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:47:14.797 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.2 2020-03-11T22:47:14.822 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:47:14.819+0000 7f5d8db12ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:14.824 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:47:14.820+0000 7f5d8db12ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:14.826 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:47:14.822+0000 7f5d8db12ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:15.176 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-03-11T22:47:15.176 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:47:15.176 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:47:15.176 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:47:15.177 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:47:15.177 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:47:15.177 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:47:15.177 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:47:15.177 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:47:15.177 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:47:15.354 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:47:15.350+0000 7f5d8db12ec0 -1 Falling back to public interface 2020-03-11T22:47:15.604 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:47:16.096 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:47:16.092+0000 7f5d8db12ec0 -1 osd.2 0 log_to_monitors {default=true} 2020-03-11T22:47:16.606 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:47:16.606 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:47:16.606 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:47:16.606 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:47:16.607 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:47:16.607 INFO:tasks.workunit.client.0.smithi059.stdout:1 2020-03-11T22:47:16.991 INFO:tasks.workunit.client.0.smithi059.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/63894,v1:127.0.0.1:6819/63894] [v2:127.0.0.1:6820/63894,v1:127.0.0.1:6821/63894] exists,up f739d588-0568-4527-ab7b-85ef8fc4e62b 2020-03-11T22:47:16.992 INFO:tasks.workunit.client.0.smithi059.stdout:add osd3 e2b46725-0201-46e6-8411-06a9f7fd475d 2020-03-11T22:47:16.992 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:47:16.992 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:47:16.992 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:47:16.992 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:47:16.992 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:59: TEST_backfill_priority: run_osd td/osd-backfill-prio 3 2020-03-11T22:47:16.992 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-prio 2020-03-11T22:47:16.993 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:47:16.993 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=3 2020-03-11T22:47:16.993 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:47:16.993 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-prio/3 2020-03-11T22:47:16.993 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=34f3141b-0483-4f39-a129-ea18fdfbfb6a --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-03-11T22:47:16.993 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:47:16.994 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:47:16.994 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:47:16.994 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/3' 2020-03-11T22:47:16.994 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/3/journal' 2020-03-11T22:47:16.994 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:47:16.994 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:47:16.994 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2020-03-11T22:47:16.995 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:47:16.995 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:47:16.995 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:47:16.995 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:47:16.995 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:47:16.995 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56462 2020-03-11T22:47:16.995 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.56462/$cluster-$name.asok' 2020-03-11T22:47:16.996 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56462/$cluster-$name.asok' 2020-03-11T22:47:16.996 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:47:16.996 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:47:16.996 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:47:16.996 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2020-03-11T22:47:16.996 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2020-03-11T22:47:16.996 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:47:16.997 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:47:16.997 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:47:16.997 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:47:16.997 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:47:16.997 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-prio/3 2020-03-11T22:47:16.997 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:47:16.997 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=e2b46725-0201-46e6-8411-06a9f7fd475d 2020-03-11T22:47:16.998 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd3 e2b46725-0201-46e6-8411-06a9f7fd475d' 2020-03-11T22:47:16.998 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:47:17.007 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQB1amlemDJHABAAdYSlg1a8qJA8Ml1d/UuD+w== 2020-03-11T22:47:17.007 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQB1amlemDJHABAAdYSlg1a8qJA8Ml1d/UuD+w=="}' 2020-03-11T22:47:17.007 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new e2b46725-0201-46e6-8411-06a9f7fd475d -i td/osd-backfill-prio/3/new.json 2020-03-11T22:47:17.550 INFO:tasks.workunit.client.0.smithi059.stdout:3 2020-03-11T22:47:17.559 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-prio/3/new.json 2020-03-11T22:47:17.560 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 3 --fsid=34f3141b-0483-4f39-a129-ea18fdfbfb6a --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/3 --osd-journal=td/osd-backfill-prio/3/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56462/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQB1amlemDJHABAAdYSlg1a8qJA8Ml1d/UuD+w== --osd-uuid e2b46725-0201-46e6-8411-06a9f7fd475d 2020-03-11T22:47:17.587 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:47:17.584+0000 7f529fb6dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:17.589 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:47:17.585+0000 7f529fb6dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:17.590 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:47:17.587+0000 7f529fb6dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:17.591 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:47:17.587+0000 7f529fb6dec0 -1 bluestore(td/osd-backfill-prio/3/block) _read_bdev_label failed to open td/osd-backfill-prio/3/block: (2) No such file or directory 2020-03-11T22:47:17.591 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:47:17.587+0000 7f529fb6dec0 -1 bluestore(td/osd-backfill-prio/3/block) _read_bdev_label failed to open td/osd-backfill-prio/3/block: (2) No such file or directory 2020-03-11T22:47:17.591 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:47:17.587+0000 7f529fb6dec0 -1 bluestore(td/osd-backfill-prio/3/block) _read_bdev_label failed to open td/osd-backfill-prio/3/block: (2) No such file or directory 2020-03-11T22:47:17.628 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:47:17.625+0000 7f529fb6dec0 -1 bluestore(td/osd-backfill-prio/3) _read_fsid unparsable uuid 2020-03-11T22:47:20.712 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-prio/3/keyring 2020-03-11T22:47:20.713 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:47:20.714 INFO:tasks.workunit.client.0.smithi059.stdout:adding osd3 key to auth repository 2020-03-11T22:47:20.714 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd3 key to auth repository 2020-03-11T22:47:20.714 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-prio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:47:21.206 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.3 2020-03-11T22:47:21.207 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.3 2020-03-11T22:47:21.207 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:47:21.207 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:47:21.207 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:47:21.207 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 3 --fsid=34f3141b-0483-4f39-a129-ea18fdfbfb6a --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/3 --osd-journal=td/osd-backfill-prio/3/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56462/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:47:21.231 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:47:21.227+0000 7fb3e954cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:21.233 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:47:21.229+0000 7fb3e954cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:21.235 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:47:21.231+0000 7fb3e954cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:21.578 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 3 2020-03-11T22:47:21.578 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:47:21.578 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=3 2020-03-11T22:47:21.578 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:47:21.578 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:47:21.578 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:47:21.579 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:47:21.579 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:47:21.579 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:47:21.579 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:47:21.750 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:47:21.746+0000 7fb3e954cec0 -1 Falling back to public interface 2020-03-11T22:47:21.943 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:47:22.520 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:47:22.516+0000 7fb3e954cec0 -1 osd.3 0 log_to_monitors {default=true} 2020-03-11T22:47:22.944 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:47:22.944 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:47:22.944 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:47:22.945 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:47:22.945 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:47:22.945 INFO:tasks.workunit.client.0.smithi059.stdout:1 2020-03-11T22:47:23.337 INFO:tasks.workunit.client.0.smithi059.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/64249,v1:127.0.0.1:6827/64249] [v2:127.0.0.1:6828/64249,v1:127.0.0.1:6829/64249] exists,up e2b46725-0201-46e6-8411-06a9f7fd475d 2020-03-11T22:47:23.337 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:47:23.337 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:47:23.337 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:47:23.338 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:47:23.338 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:59: TEST_backfill_priority: run_osd td/osd-backfill-prio 4 2020-03-11T22:47:23.338 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-prio 2020-03-11T22:47:23.338 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:47:23.338 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=4 2020-03-11T22:47:23.338 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:47:23.338 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-prio/4 2020-03-11T22:47:23.339 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=34f3141b-0483-4f39-a129-ea18fdfbfb6a --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-03-11T22:47:23.339 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:47:23.339 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:47:23.339 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:47:23.339 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/4' 2020-03-11T22:47:23.339 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/4/journal' 2020-03-11T22:47:23.339 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:47:23.340 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:47:23.340 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2020-03-11T22:47:23.340 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:47:23.340 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:47:23.340 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:47:23.340 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:47:23.340 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:47:23.341 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56462 2020-03-11T22:47:23.341 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.56462/$cluster-$name.asok' 2020-03-11T22:47:23.343 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56462/$cluster-$name.asok' 2020-03-11T22:47:23.343 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:47:23.343 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:47:23.343 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:47:23.343 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2020-03-11T22:47:23.343 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2020-03-11T22:47:23.344 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:47:23.344 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:47:23.344 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:47:23.344 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:47:23.344 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:47:23.344 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-prio/4 2020-03-11T22:47:23.344 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:47:23.345 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=aa76b8fc-f155-41f0-b89a-bbf0ca74d4a3 2020-03-11T22:47:23.345 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd4 aa76b8fc-f155-41f0-b89a-bbf0ca74d4a3' 2020-03-11T22:47:23.345 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:47:23.345 INFO:tasks.workunit.client.0.smithi059.stdout:add osd4 aa76b8fc-f155-41f0-b89a-bbf0ca74d4a3 2020-03-11T22:47:23.356 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQB7amlewSEQFRAAJshow6JT+gQwnIy5JsJAkA== 2020-03-11T22:47:23.356 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQB7amlewSEQFRAAJshow6JT+gQwnIy5JsJAkA=="}' 2020-03-11T22:47:23.356 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new aa76b8fc-f155-41f0-b89a-bbf0ca74d4a3 -i td/osd-backfill-prio/4/new.json 2020-03-11T22:47:23.848 INFO:tasks.workunit.client.0.smithi059.stdout:4 2020-03-11T22:47:23.858 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-prio/4/new.json 2020-03-11T22:47:23.859 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 4 --fsid=34f3141b-0483-4f39-a129-ea18fdfbfb6a --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/4 --osd-journal=td/osd-backfill-prio/4/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56462/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQB7amlewSEQFRAAJshow6JT+gQwnIy5JsJAkA== --osd-uuid aa76b8fc-f155-41f0-b89a-bbf0ca74d4a3 2020-03-11T22:47:23.887 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:47:23.883+0000 7fb8861faec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:23.889 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:47:23.885+0000 7fb8861faec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:23.889 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:47:23.886+0000 7fb8861faec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:23.890 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:47:23.886+0000 7fb8861faec0 -1 bluestore(td/osd-backfill-prio/4/block) _read_bdev_label failed to open td/osd-backfill-prio/4/block: (2) No such file or directory 2020-03-11T22:47:23.890 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:47:23.886+0000 7fb8861faec0 -1 bluestore(td/osd-backfill-prio/4/block) _read_bdev_label failed to open td/osd-backfill-prio/4/block: (2) No such file or directory 2020-03-11T22:47:23.891 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:47:23.886+0000 7fb8861faec0 -1 bluestore(td/osd-backfill-prio/4/block) _read_bdev_label failed to open td/osd-backfill-prio/4/block: (2) No such file or directory 2020-03-11T22:47:23.928 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:47:23.924+0000 7fb8861faec0 -1 bluestore(td/osd-backfill-prio/4) _read_fsid unparsable uuid 2020-03-11T22:47:27.290 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-prio/4/keyring 2020-03-11T22:47:27.290 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:47:27.291 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd4 key to auth repository 2020-03-11T22:47:27.291 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-prio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:47:27.292 INFO:tasks.workunit.client.0.smithi059.stdout:adding osd4 key to auth repository 2020-03-11T22:47:27.781 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.4 2020-03-11T22:47:27.781 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:47:27.782 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 4 --fsid=34f3141b-0483-4f39-a129-ea18fdfbfb6a --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/4 --osd-journal=td/osd-backfill-prio/4/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56462/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:47:27.782 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:47:27.782 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:47:27.782 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.4 2020-03-11T22:47:27.806 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:47:27.802+0000 7f309594dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:27.808 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:47:27.804+0000 7f309594dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:27.810 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:47:27.806+0000 7f309594dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:28.155 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 4 2020-03-11T22:47:28.155 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:47:28.156 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=4 2020-03-11T22:47:28.156 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:47:28.156 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:47:28.156 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:47:28.156 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:47:28.156 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:47:28.156 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.4 up' 2020-03-11T22:47:28.157 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:47:28.337 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:47:28.333+0000 7f309594dec0 -1 Falling back to public interface 2020-03-11T22:47:28.542 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:47:29.087 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:47:29.083+0000 7f309594dec0 -1 osd.4 0 log_to_monitors {default=true} 2020-03-11T22:47:29.543 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:47:29.543 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:47:29.544 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:47:29.544 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:47:29.544 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.4 up' 2020-03-11T22:47:29.544 INFO:tasks.workunit.client.0.smithi059.stdout:1 2020-03-11T22:47:29.946 INFO:tasks.workunit.client.0.smithi059.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/64604,v1:127.0.0.1:6835/64604] [v2:127.0.0.1:6836/64604,v1:127.0.0.1:6837/64604] exists,up aa76b8fc-f155-41f0-b89a-bbf0ca74d4a3 2020-03-11T22:47:29.947 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:47:29.947 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:47:29.947 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:47:29.947 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: seq 1 10 2020-03-11T22:47:29.948 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:47:29.948 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test1 1 1 2020-03-11T22:47:29.948 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test1 1 1 2020-03-11T22:47:30.597 INFO:tasks.workunit.client.0.smithi059.stderr:pool 'test1' already exists 2020-03-11T22:47:30.605 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:47:31.606 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test1 size 2 2020-03-11T22:47:32.444 INFO:tasks.workunit.client.0.smithi059.stderr:set pool 1 size to 2 2020-03-11T22:47:32.453 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:47:32.453 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test2 1 1 2020-03-11T22:47:32.453 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test2 1 1 2020-03-11T22:47:32.994 INFO:tasks.workunit.client.0.smithi059.stderr:pool 'test2' already exists 2020-03-11T22:47:33.002 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:47:34.003 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test2 size 2 2020-03-11T22:47:34.716 INFO:tasks.workunit.client.0.smithi059.stderr:set pool 2 size to 2 2020-03-11T22:47:34.724 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:47:34.724 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test3 1 1 2020-03-11T22:47:34.725 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test3 1 1 2020-03-11T22:47:35.528 INFO:tasks.workunit.client.0.smithi059.stderr:pool 'test3' already exists 2020-03-11T22:47:35.536 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:47:36.537 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test3 size 2 2020-03-11T22:47:37.222 INFO:tasks.workunit.client.0.smithi059.stderr:set pool 3 size to 2 2020-03-11T22:47:37.233 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:47:37.233 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test4 1 1 2020-03-11T22:47:37.233 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test4 1 1 2020-03-11T22:47:37.846 INFO:tasks.workunit.client.0.smithi059.stderr:pool 'test4' already exists 2020-03-11T22:47:37.853 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:47:38.855 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test4 size 2 2020-03-11T22:47:39.595 INFO:tasks.workunit.client.0.smithi059.stderr:set pool 4 size to 2 2020-03-11T22:47:39.603 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:47:39.603 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test5 1 1 2020-03-11T22:47:39.603 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test5 1 1 2020-03-11T22:47:40.200 INFO:tasks.workunit.client.0.smithi059.stderr:pool 'test5' already exists 2020-03-11T22:47:40.207 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:47:41.209 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test5 size 2 2020-03-11T22:47:42.010 INFO:tasks.workunit.client.0.smithi059.stderr:set pool 5 size to 2 2020-03-11T22:47:42.018 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:47:42.019 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test6 1 1 2020-03-11T22:47:42.019 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test6 1 1 2020-03-11T22:47:42.642 INFO:tasks.workunit.client.0.smithi059.stderr:pool 'test6' already exists 2020-03-11T22:47:42.650 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:47:43.651 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test6 size 2 2020-03-11T22:47:44.350 INFO:tasks.workunit.client.0.smithi059.stderr:set pool 6 size to 2 2020-03-11T22:47:44.358 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:47:44.358 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test7 1 1 2020-03-11T22:47:44.358 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test7 1 1 2020-03-11T22:47:44.923 INFO:tasks.workunit.client.0.smithi059.stderr:pool 'test7' already exists 2020-03-11T22:47:44.931 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:47:45.932 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test7 size 2 2020-03-11T22:47:46.822 INFO:tasks.workunit.client.0.smithi059.stderr:set pool 7 size to 2 2020-03-11T22:47:46.831 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:47:46.831 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test8 1 1 2020-03-11T22:47:46.831 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test8 1 1 2020-03-11T22:47:47.384 INFO:tasks.workunit.client.0.smithi059.stderr:pool 'test8' already exists 2020-03-11T22:47:47.392 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:47:48.393 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test8 size 2 2020-03-11T22:47:49.242 INFO:tasks.workunit.client.0.smithi059.stderr:set pool 8 size to 2 2020-03-11T22:47:49.250 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:47:49.250 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test9 1 1 2020-03-11T22:47:49.250 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test9 1 1 2020-03-11T22:47:49.883 INFO:tasks.workunit.client.0.smithi059.stderr:pool 'test9' already exists 2020-03-11T22:47:49.891 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:47:50.892 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test9 size 2 2020-03-11T22:47:51.811 INFO:tasks.workunit.client.0.smithi059.stderr:set pool 9 size to 2 2020-03-11T22:47:51.819 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:47:51.820 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test10 1 1 2020-03-11T22:47:51.820 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test10 1 1 2020-03-11T22:47:52.464 INFO:tasks.workunit.client.0.smithi059.stderr:pool 'test10' already exists 2020-03-11T22:47:52.471 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:47:53.473 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test10 size 2 2020-03-11T22:47:54.142 INFO:tasks.workunit.client.0.smithi059.stderr:set pool 10 size to 2 2020-03-11T22:47:54.150 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:67: TEST_backfill_priority: sleep 5 2020-03-11T22:47:59.153 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:69: TEST_backfill_priority: wait_for_clean 2020-03-11T22:47:59.153 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:47:59.153 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:47:59.154 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:47:59.154 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:47:59.154 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:47:59.154 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:47:59.154 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:47:59.155 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:47:59.155 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:47:59.155 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:47:59.259 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:47:59.259 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:47:59.259 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:47:59.260 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:47:59.260 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:47:59.630 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:47:59.630 INFO:tasks.workunit.client.0.smithi059.stderr:1 2020-03-11T22:47:59.631 INFO:tasks.workunit.client.0.smithi059.stderr:2 2020-03-11T22:47:59.631 INFO:tasks.workunit.client.0.smithi059.stderr:3 2020-03-11T22:47:59.631 INFO:tasks.workunit.client.0.smithi059.stderr:4' 2020-03-11T22:47:59.631 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:47:59.632 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:47:59.632 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:47:59.787 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836494 2020-03-11T22:47:59.788 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836494' 2020-03-11T22:47:59.788 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:47:59.788 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:47:59.919 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672972 2020-03-11T22:47:59.919 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836494 1-42949672972' 2020-03-11T22:47:59.919 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:47:59.920 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:48:00.051 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509452 2020-03-11T22:48:00.052 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836494 1-42949672972 2-64424509452' 2020-03-11T22:48:00.052 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:48:00.052 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:48:00.184 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=85899345930 2020-03-11T22:48:00.184 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836494 1-42949672972 2-64424509452 3-85899345930' 2020-03-11T22:48:00.184 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:48:00.184 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:48:00.316 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=107374182408 2020-03-11T22:48:00.316 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836494 1-42949672972 2-64424509452 3-85899345930 4-107374182408' 2020-03-11T22:48:00.316 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:48:00.317 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836494 2020-03-11T22:48:00.317 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:48:00.318 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:48:00.318 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836494 2020-03-11T22:48:00.318 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:48:00.319 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836494 2020-03-11T22:48:00.319 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836494' 2020-03-11T22:48:00.319 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:48:00.319 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.0 seq 21474836494 2020-03-11T22:48:00.683 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836491 -lt 21474836494 2020-03-11T22:48:00.684 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:48:01.684 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:48:01.685 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:48:02.072 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836494 -lt 21474836494 2020-03-11T22:48:02.072 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:48:02.073 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672972 2020-03-11T22:48:02.073 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:48:02.073 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:48:02.074 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672972 2020-03-11T22:48:02.074 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:48:02.074 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672972 2020-03-11T22:48:02.075 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2020-03-11T22:48:02.075 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.1 seq 42949672972 2020-03-11T22:48:02.075 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:48:02.440 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672973 -lt 42949672972 2020-03-11T22:48:02.440 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:48:02.441 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509452 2020-03-11T22:48:02.441 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:48:02.441 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:48:02.442 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509452 2020-03-11T22:48:02.442 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:48:02.443 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509452 2020-03-11T22:48:02.443 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2020-03-11T22:48:02.443 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:48:02.443 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.2 seq 64424509452 2020-03-11T22:48:02.809 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509452 -lt 64424509452 2020-03-11T22:48:02.810 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:48:02.810 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-85899345930 2020-03-11T22:48:02.810 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:48:02.811 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:48:02.811 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-85899345930 2020-03-11T22:48:02.812 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:48:02.812 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=85899345930 2020-03-11T22:48:02.812 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 85899345930' 2020-03-11T22:48:02.813 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:48:02.813 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.3 seq 85899345930 2020-03-11T22:48:03.180 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 85899345930 -lt 85899345930 2020-03-11T22:48:03.180 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:48:03.181 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-107374182408 2020-03-11T22:48:03.181 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:48:03.182 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:48:03.182 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-107374182408 2020-03-11T22:48:03.182 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:48:03.183 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=107374182408 2020-03-11T22:48:03.183 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 107374182408' 2020-03-11T22:48:03.184 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:48:03.184 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.4 seq 107374182408 2020-03-11T22:48:03.578 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 107374182409 -lt 107374182408 2020-03-11T22:48:03.578 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:48:03.579 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:48:03.579 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:48:04.063 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 10 == 0 2020-03-11T22:48:04.063 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:48:04.064 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:48:04.064 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:48:04.064 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:48:04.064 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:48:04.064 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:48:04.064 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:48:04.418 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=10 2020-03-11T22:48:04.419 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:48:04.419 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:48:04.419 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:48:04.892 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 10 = 10 2020-03-11T22:48:04.892 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:48:04.892 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:48:04.893 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:71: TEST_backfill_priority: ceph pg dump pgs 2020-03-11T22:48:05.235 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:48:05.235 INFO:tasks.workunit.client.0.smithi059.stdout:10.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:47:55.511138+0000 0'0 83:15 [2,1] 2 [2,1] 2 0'0 2020-03-11T22:47:52.199698+0000 0'0 2020-03-11T22:47:52.199698+0000 0 2020-03-11T22:48:05.235 INFO:tasks.workunit.client.0.smithi059.stdout:9.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:47:52.329002+0000 0'0 83:27 [1,2] 1 [1,2] 1 0'0 2020-03-11T22:47:49.665021+0000 0'0 2020-03-11T22:47:49.665021+0000 0 2020-03-11T22:48:05.235 INFO:tasks.workunit.client.0.smithi059.stdout:8.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:47:49.564759+0000 0'0 83:29 [3,4] 3 [3,4] 3 0'0 2020-03-11T22:47:47.175918+0000 0'0 2020-03-11T22:47:47.175918+0000 0 2020-03-11T22:48:05.236 INFO:tasks.workunit.client.0.smithi059.stdout:7.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:47:47.427611+0000 0'0 83:30 [1,4] 1 [1,4] 1 0'0 2020-03-11T22:47:44.646052+0000 0'0 2020-03-11T22:47:44.646052+0000 0 2020-03-11T22:48:05.236 INFO:tasks.workunit.client.0.smithi059.stdout:2.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:47:35.280308+0000 0'0 83:45 [3,1] 3 [3,1] 3 0'0 2020-03-11T22:47:32.764982+0000 0'0 2020-03-11T22:47:32.764982+0000 0 2020-03-11T22:48:05.236 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:47:33.257226+0000 0'0 83:59 [1,0] 1 [1,0] 1 0'0 2020-03-11T22:47:30.333509+0000 0'0 2020-03-11T22:47:30.333509+0000 0 2020-03-11T22:48:05.236 INFO:tasks.workunit.client.0.smithi059.stdout:3.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:47:37.801015+0000 0'0 83:52 [1,2] 1 [1,2] 1 0'0 2020-03-11T22:47:35.075766+0000 0'0 2020-03-11T22:47:35.075766+0000 0 2020-03-11T22:48:05.236 INFO:tasks.workunit.client.0.smithi059.stdout:4.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:47:40.261480+0000 0'0 83:49 [4,0] 4 [4,0] 4 0'0 2020-03-11T22:47:37.632320+0000 0'0 2020-03-11T22:47:37.632320+0000 0 2020-03-11T22:48:05.236 INFO:tasks.workunit.client.0.smithi059.stdout:5.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:47:42.319489+0000 0'0 83:39 [3,4] 3 [3,4] 3 0'0 2020-03-11T22:47:40.018236+0000 0'0 2020-03-11T22:47:40.018236+0000 0 2020-03-11T22:48:05.237 INFO:tasks.workunit.client.0.smithi059.stdout:6.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:47:44.863169+0000 0'0 83:40 [4,2] 4 [4,2] 4 0'0 2020-03-11T22:47:42.389586+0000 0'0 2020-03-11T22:47:42.389586+0000 0 2020-03-11T22:48:05.237 INFO:tasks.workunit.client.0.smithi059.stdout: 2020-03-11T22:48:05.237 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:48:05.237 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2020-03-11T22:48:05.245 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:75: TEST_backfill_priority: local PG1 2020-03-11T22:48:05.246 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:76: TEST_backfill_priority: local POOLNUM1 2020-03-11T22:48:05.246 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:77: TEST_backfill_priority: local pool1 2020-03-11T22:48:05.246 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:78: TEST_backfill_priority: local chk_osd1_1 2020-03-11T22:48:05.246 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:79: TEST_backfill_priority: local chk_osd1_2 2020-03-11T22:48:05.246 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:81: TEST_backfill_priority: local PG2 2020-03-11T22:48:05.246 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:82: TEST_backfill_priority: local POOLNUM2 2020-03-11T22:48:05.247 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:83: TEST_backfill_priority: local pool2 2020-03-11T22:48:05.247 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:84: TEST_backfill_priority: local chk_osd2 2020-03-11T22:48:05.247 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:86: TEST_backfill_priority: local PG3 2020-03-11T22:48:05.247 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:87: TEST_backfill_priority: local POOLNUM3 2020-03-11T22:48:05.247 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:88: TEST_backfill_priority: local pool3 2020-03-11T22:48:05.247 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: seq 1 10 2020-03-11T22:48:05.247 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:48:05.248 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 1.0 --format=json 2020-03-11T22:48:05.248 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2020-03-11T22:48:05.604 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2020-03-11T22:48:05.604 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=1 2020-03-11T22:48:05.605 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2020-03-11T22:48:05.606 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=0 2020-03-11T22:48:05.606 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z '' ']' 2020-03-11T22:48:05.606 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:97: TEST_backfill_priority: PG1=1.0 2020-03-11T22:48:05.606 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:98: TEST_backfill_priority: POOLNUM1=1 2020-03-11T22:48:05.606 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:99: TEST_backfill_priority: pool1=test1 2020-03-11T22:48:05.607 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:100: TEST_backfill_priority: chk_osd1_1=1 2020-03-11T22:48:05.607 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:101: TEST_backfill_priority: chk_osd1_2=0 2020-03-11T22:48:05.607 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:48:05.607 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 2.0 --format=json 2020-03-11T22:48:05.607 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2020-03-11T22:48:05.974 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2020-03-11T22:48:05.975 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=3 2020-03-11T22:48:05.975 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2020-03-11T22:48:05.976 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=1 2020-03-11T22:48:05.976 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z 1.0 ']' 2020-03-11T22:48:05.976 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:102: TEST_backfill_priority: '[' -z '' -a 1 = 3 -a 0 '!=' 1 ']' 2020-03-11T22:48:05.976 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:108: TEST_backfill_priority: '[' -n '' -a 1 = 3 -a 0 '!=' 1 -a '' '!=' 1 ']' 2020-03-11T22:48:05.977 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:48:05.977 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 3.0 --format=json 2020-03-11T22:48:05.977 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2020-03-11T22:48:06.356 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2020-03-11T22:48:06.356 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=1 2020-03-11T22:48:06.357 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2020-03-11T22:48:06.358 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=2 2020-03-11T22:48:06.358 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z 1.0 ']' 2020-03-11T22:48:06.358 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:102: TEST_backfill_priority: '[' -z '' -a 1 = 1 -a 0 '!=' 2 ']' 2020-03-11T22:48:06.359 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:104: TEST_backfill_priority: PG2=3.0 2020-03-11T22:48:06.359 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:105: TEST_backfill_priority: POOLNUM2=3 2020-03-11T22:48:06.359 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:106: TEST_backfill_priority: pool2=test3 2020-03-11T22:48:06.359 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:107: TEST_backfill_priority: chk_osd2=2 2020-03-11T22:48:06.359 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:48:06.359 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 4.0 --format=json 2020-03-11T22:48:06.359 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2020-03-11T22:48:06.717 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2020-03-11T22:48:06.717 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=4 2020-03-11T22:48:06.718 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2020-03-11T22:48:06.719 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=0 2020-03-11T22:48:06.719 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z 1.0 ']' 2020-03-11T22:48:06.719 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:102: TEST_backfill_priority: '[' -z 3.0 -a 1 = 4 -a 0 '!=' 0 ']' 2020-03-11T22:48:06.719 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:108: TEST_backfill_priority: '[' -n 3.0 -a 1 = 4 -a 0 '!=' 0 -a 2 '!=' 0 ']' 2020-03-11T22:48:06.719 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:48:06.720 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 5.0 --format=json 2020-03-11T22:48:06.720 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2020-03-11T22:48:07.102 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2020-03-11T22:48:07.103 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=3 2020-03-11T22:48:07.103 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2020-03-11T22:48:07.104 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=4 2020-03-11T22:48:07.105 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z 1.0 ']' 2020-03-11T22:48:07.105 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:102: TEST_backfill_priority: '[' -z 3.0 -a 1 = 3 -a 0 '!=' 4 ']' 2020-03-11T22:48:07.105 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:108: TEST_backfill_priority: '[' -n 3.0 -a 1 = 3 -a 0 '!=' 4 -a 2 '!=' 4 ']' 2020-03-11T22:48:07.105 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:48:07.105 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 6.0 --format=json 2020-03-11T22:48:07.105 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2020-03-11T22:48:07.520 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2020-03-11T22:48:07.521 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=4 2020-03-11T22:48:07.521 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2020-03-11T22:48:07.522 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=2 2020-03-11T22:48:07.522 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z 1.0 ']' 2020-03-11T22:48:07.522 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:102: TEST_backfill_priority: '[' -z 3.0 -a 1 = 4 -a 0 '!=' 2 ']' 2020-03-11T22:48:07.523 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:108: TEST_backfill_priority: '[' -n 3.0 -a 1 = 4 -a 0 '!=' 2 -a 2 '!=' 2 ']' 2020-03-11T22:48:07.523 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:48:07.523 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 7.0 --format=json 2020-03-11T22:48:07.523 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2020-03-11T22:48:07.883 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2020-03-11T22:48:07.884 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=1 2020-03-11T22:48:07.884 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2020-03-11T22:48:07.886 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=4 2020-03-11T22:48:07.886 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z 1.0 ']' 2020-03-11T22:48:07.886 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:102: TEST_backfill_priority: '[' -z 3.0 -a 1 = 1 -a 0 '!=' 4 ']' 2020-03-11T22:48:07.886 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:108: TEST_backfill_priority: '[' -n 3.0 -a 1 = 1 -a 0 '!=' 4 -a 2 '!=' 4 ']' 2020-03-11T22:48:07.886 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:110: TEST_backfill_priority: PG3=7.0 2020-03-11T22:48:07.886 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:111: TEST_backfill_priority: POOLNUM3=7 2020-03-11T22:48:07.887 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:112: TEST_backfill_priority: pool3=test7 2020-03-11T22:48:07.887 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:113: TEST_backfill_priority: break 2020-03-11T22:48:07.887 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:116: TEST_backfill_priority: rm -f td/osd-backfill-prio/acting 2020-03-11T22:48:07.887 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:118: TEST_backfill_priority: '[' test3 = '' -o pool3 = '' ']' 2020-03-11T22:48:07.888 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: seq 1 10 2020-03-11T22:48:07.888 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:48:07.888 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 1 '!=' 1 -a 1 '!=' 3 -a 1 '!=' 7 ']' 2020-03-11T22:48:07.889 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:48:07.889 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 2 '!=' 1 -a 2 '!=' 3 -a 2 '!=' 7 ']' 2020-03-11T22:48:07.889 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test2 2020-03-11T22:48:07.889 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test2 2020-03-11T22:48:07.889 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test2 test2 --yes-i-really-really-mean-it 2020-03-11T22:48:08.383 INFO:tasks.workunit.client.0.smithi059.stderr:pool 'test2' does not exist 2020-03-11T22:48:08.392 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:48:08.392 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 3 '!=' 1 -a 3 '!=' 3 -a 3 '!=' 7 ']' 2020-03-11T22:48:08.392 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:48:08.392 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 4 '!=' 1 -a 4 '!=' 3 -a 4 '!=' 7 ']' 2020-03-11T22:48:08.392 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test4 2020-03-11T22:48:08.392 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test4 2020-03-11T22:48:08.393 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test4 test4 --yes-i-really-really-mean-it 2020-03-11T22:48:08.911 INFO:tasks.workunit.client.0.smithi059.stderr:pool 'test4' does not exist 2020-03-11T22:48:08.919 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:48:08.919 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 5 '!=' 1 -a 5 '!=' 3 -a 5 '!=' 7 ']' 2020-03-11T22:48:08.919 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test5 2020-03-11T22:48:08.919 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test5 2020-03-11T22:48:08.919 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test5 test5 --yes-i-really-really-mean-it 2020-03-11T22:48:09.524 INFO:tasks.workunit.client.0.smithi059.stderr:pool 'test5' does not exist 2020-03-11T22:48:09.533 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:48:09.533 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 6 '!=' 1 -a 6 '!=' 3 -a 6 '!=' 7 ']' 2020-03-11T22:48:09.533 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test6 2020-03-11T22:48:09.533 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test6 2020-03-11T22:48:09.533 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test6 test6 --yes-i-really-really-mean-it 2020-03-11T22:48:10.081 INFO:tasks.workunit.client.0.smithi059.stderr:pool 'test6' does not exist 2020-03-11T22:48:10.089 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:48:10.089 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 7 '!=' 1 -a 7 '!=' 3 -a 7 '!=' 7 ']' 2020-03-11T22:48:10.089 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:48:10.089 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 8 '!=' 1 -a 8 '!=' 3 -a 8 '!=' 7 ']' 2020-03-11T22:48:10.089 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test8 2020-03-11T22:48:10.090 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test8 2020-03-11T22:48:10.090 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test8 test8 --yes-i-really-really-mean-it 2020-03-11T22:48:10.677 INFO:tasks.workunit.client.0.smithi059.stderr:pool 'test8' does not exist 2020-03-11T22:48:10.685 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:48:10.686 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 9 '!=' 1 -a 9 '!=' 3 -a 9 '!=' 7 ']' 2020-03-11T22:48:10.686 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test9 2020-03-11T22:48:10.686 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test9 2020-03-11T22:48:10.686 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test9 test9 --yes-i-really-really-mean-it 2020-03-11T22:48:11.305 INFO:tasks.workunit.client.0.smithi059.stderr:pool 'test9' does not exist 2020-03-11T22:48:11.313 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:48:11.314 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 10 '!=' 1 -a 10 '!=' 3 -a 10 '!=' 7 ']' 2020-03-11T22:48:11.314 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test10 2020-03-11T22:48:11.314 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test10 2020-03-11T22:48:11.314 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test10 test10 --yes-i-really-really-mean-it 2020-03-11T22:48:11.884 INFO:tasks.workunit.client.0.smithi059.stderr:pool 'test10' does not exist 2020-03-11T22:48:11.892 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:132: TEST_backfill_priority: ceph osd pool set test3 size 1 2020-03-11T22:48:12.909 INFO:tasks.workunit.client.0.smithi059.stderr:set pool 3 size to 1 2020-03-11T22:48:12.917 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:133: TEST_backfill_priority: ceph osd pool set test7 size 1 2020-03-11T22:48:13.736 INFO:tasks.workunit.client.0.smithi059.stderr:set pool 7 size to 1 2020-03-11T22:48:13.744 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:134: TEST_backfill_priority: wait_for_clean 2020-03-11T22:48:13.745 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:48:13.745 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:48:13.745 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:48:13.745 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:48:13.745 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:48:13.746 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:48:13.746 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:48:13.746 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:48:13.746 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:48:13.747 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:48:13.840 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:48:13.840 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:48:13.840 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:48:13.840 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:48:13.841 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:48:14.210 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:48:14.210 INFO:tasks.workunit.client.0.smithi059.stderr:1 2020-03-11T22:48:14.211 INFO:tasks.workunit.client.0.smithi059.stderr:2 2020-03-11T22:48:14.211 INFO:tasks.workunit.client.0.smithi059.stderr:3 2020-03-11T22:48:14.211 INFO:tasks.workunit.client.0.smithi059.stderr:4' 2020-03-11T22:48:14.211 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:48:14.211 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:48:14.211 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:48:14.344 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836498 2020-03-11T22:48:14.344 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836498' 2020-03-11T22:48:14.344 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:48:14.344 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:48:14.478 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672977 2020-03-11T22:48:14.479 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836498 1-42949672977' 2020-03-11T22:48:14.479 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:48:14.479 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:48:14.636 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509457 2020-03-11T22:48:14.637 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836498 1-42949672977 2-64424509457' 2020-03-11T22:48:14.637 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:48:14.637 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:48:14.769 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=85899345935 2020-03-11T22:48:14.769 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836498 1-42949672977 2-64424509457 3-85899345935' 2020-03-11T22:48:14.769 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:48:14.770 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:48:14.901 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=107374182413 2020-03-11T22:48:14.901 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836498 1-42949672977 2-64424509457 3-85899345935 4-107374182413' 2020-03-11T22:48:14.902 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:48:14.902 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836498 2020-03-11T22:48:14.902 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:48:14.903 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:48:14.903 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836498 2020-03-11T22:48:14.903 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:48:14.904 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836498 2020-03-11T22:48:14.904 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2020-03-11T22:48:14.904 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:48:14.905 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.0 seq 21474836498 2020-03-11T22:48:15.279 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836496 -lt 21474836498 2020-03-11T22:48:15.279 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:48:16.280 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:48:16.280 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:48:16.661 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836499 -lt 21474836498 2020-03-11T22:48:16.661 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:48:16.661 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672977 2020-03-11T22:48:16.662 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:48:16.662 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:48:16.663 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672977 2020-03-11T22:48:16.663 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:48:16.664 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672977 2020-03-11T22:48:16.664 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2020-03-11T22:48:16.664 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:48:16.664 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.1 seq 42949672977 2020-03-11T22:48:17.028 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672977 -lt 42949672977 2020-03-11T22:48:17.028 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:48:17.028 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509457 2020-03-11T22:48:17.029 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:48:17.029 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:48:17.030 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509457 2020-03-11T22:48:17.030 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:48:17.031 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509457 2020-03-11T22:48:17.031 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2020-03-11T22:48:17.031 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:48:17.031 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.2 seq 64424509457 2020-03-11T22:48:17.409 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509457 -lt 64424509457 2020-03-11T22:48:17.410 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:48:17.410 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-85899345935 2020-03-11T22:48:17.410 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:48:17.411 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:48:17.411 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-85899345935 2020-03-11T22:48:17.412 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:48:17.412 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=85899345935 2020-03-11T22:48:17.412 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 85899345935' 2020-03-11T22:48:17.413 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:48:17.413 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.3 seq 85899345935 2020-03-11T22:48:17.798 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 85899345935 -lt 85899345935 2020-03-11T22:48:17.798 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:48:17.799 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-107374182413 2020-03-11T22:48:17.799 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:48:17.799 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:48:17.800 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-107374182413 2020-03-11T22:48:17.800 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:48:17.801 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=107374182413 2020-03-11T22:48:17.801 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 107374182413' 2020-03-11T22:48:17.801 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:48:17.801 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.4 seq 107374182413 2020-03-11T22:48:18.182 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 107374182414 -lt 107374182413 2020-03-11T22:48:18.182 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:48:18.183 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:48:18.183 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:48:18.673 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 3 == 0 2020-03-11T22:48:18.673 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:48:18.673 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:48:18.673 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:48:18.674 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:48:18.674 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:48:18.674 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:48:18.674 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:48:19.025 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=3 2020-03-11T22:48:19.026 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:48:19.026 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:48:19.026 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:48:19.499 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 3 = 3 2020-03-11T22:48:19.499 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:48:19.500 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:48:19.500 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:136: TEST_backfill_priority: dd if=/dev/urandom of=td/osd-backfill-prio/data bs=1M count=10 2020-03-11T22:48:19.546 INFO:tasks.workunit.client.0.smithi059.stderr:10+0 records in 2020-03-11T22:48:19.546 INFO:tasks.workunit.client.0.smithi059.stderr:10+0 records out 2020-03-11T22:48:19.547 INFO:tasks.workunit.client.0.smithi059.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0462901 s, 227 MB/s 2020-03-11T22:48:19.547 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:137: TEST_backfill_priority: p=1 2020-03-11T22:48:19.547 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:138: TEST_backfill_priority: for pname in $pool1 $pool2 $pool3 2020-03-11T22:48:19.548 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: seq 1 50 2020-03-11T22:48:19.548 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:19.548 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj1-p1 td/osd-backfill-prio/data 2020-03-11T22:48:20.068 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:20.069 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj2-p1 td/osd-backfill-prio/data 2020-03-11T22:48:20.470 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:20.470 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj3-p1 td/osd-backfill-prio/data 2020-03-11T22:48:20.961 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:20.961 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj4-p1 td/osd-backfill-prio/data 2020-03-11T22:48:21.473 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:21.473 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj5-p1 td/osd-backfill-prio/data 2020-03-11T22:48:21.924 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:21.924 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj6-p1 td/osd-backfill-prio/data 2020-03-11T22:48:22.417 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:22.417 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj7-p1 td/osd-backfill-prio/data 2020-03-11T22:48:22.910 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:22.910 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj8-p1 td/osd-backfill-prio/data 2020-03-11T22:48:23.354 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:23.355 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj9-p1 td/osd-backfill-prio/data 2020-03-11T22:48:23.845 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:23.845 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj10-p1 td/osd-backfill-prio/data 2020-03-11T22:48:24.397 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:24.398 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj11-p1 td/osd-backfill-prio/data 2020-03-11T22:48:24.803 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:24.804 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj12-p1 td/osd-backfill-prio/data 2020-03-11T22:48:25.308 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:25.309 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj13-p1 td/osd-backfill-prio/data 2020-03-11T22:48:25.795 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:25.795 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj14-p1 td/osd-backfill-prio/data 2020-03-11T22:48:26.237 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:26.237 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj15-p1 td/osd-backfill-prio/data 2020-03-11T22:48:26.754 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:26.754 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj16-p1 td/osd-backfill-prio/data 2020-03-11T22:48:27.190 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:27.190 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj17-p1 td/osd-backfill-prio/data 2020-03-11T22:48:27.729 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:27.729 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj18-p1 td/osd-backfill-prio/data 2020-03-11T22:48:28.242 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:28.243 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj19-p1 td/osd-backfill-prio/data 2020-03-11T22:48:28.762 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:28.763 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj20-p1 td/osd-backfill-prio/data 2020-03-11T22:48:29.187 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:29.187 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj21-p1 td/osd-backfill-prio/data 2020-03-11T22:48:29.730 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:29.730 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj22-p1 td/osd-backfill-prio/data 2020-03-11T22:48:30.129 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:30.130 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj23-p1 td/osd-backfill-prio/data 2020-03-11T22:48:30.682 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:30.682 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj24-p1 td/osd-backfill-prio/data 2020-03-11T22:48:31.079 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:31.080 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj25-p1 td/osd-backfill-prio/data 2020-03-11T22:48:31.626 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:31.627 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj26-p1 td/osd-backfill-prio/data 2020-03-11T22:48:32.063 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:32.063 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj27-p1 td/osd-backfill-prio/data 2020-03-11T22:48:32.721 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:32.722 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj28-p1 td/osd-backfill-prio/data 2020-03-11T22:48:33.196 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:33.197 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj29-p1 td/osd-backfill-prio/data 2020-03-11T22:48:33.690 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:33.690 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj30-p1 td/osd-backfill-prio/data 2020-03-11T22:48:34.163 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:34.164 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj31-p1 td/osd-backfill-prio/data 2020-03-11T22:48:34.642 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:34.643 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj32-p1 td/osd-backfill-prio/data 2020-03-11T22:48:35.105 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:35.105 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj33-p1 td/osd-backfill-prio/data 2020-03-11T22:48:35.588 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:35.589 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj34-p1 td/osd-backfill-prio/data 2020-03-11T22:48:36.079 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:36.080 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj35-p1 td/osd-backfill-prio/data 2020-03-11T22:48:36.514 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:36.514 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj36-p1 td/osd-backfill-prio/data 2020-03-11T22:48:37.032 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:37.032 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj37-p1 td/osd-backfill-prio/data 2020-03-11T22:48:37.608 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:37.609 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj38-p1 td/osd-backfill-prio/data 2020-03-11T22:48:38.119 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:38.119 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj39-p1 td/osd-backfill-prio/data 2020-03-11T22:48:38.611 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:38.612 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj40-p1 td/osd-backfill-prio/data 2020-03-11T22:48:39.079 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:39.079 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj41-p1 td/osd-backfill-prio/data 2020-03-11T22:48:39.475 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:39.475 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj42-p1 td/osd-backfill-prio/data 2020-03-11T22:48:40.006 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:40.007 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj43-p1 td/osd-backfill-prio/data 2020-03-11T22:48:40.431 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:40.431 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj44-p1 td/osd-backfill-prio/data 2020-03-11T22:48:40.906 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:40.907 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj45-p1 td/osd-backfill-prio/data 2020-03-11T22:48:41.440 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:41.440 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj46-p1 td/osd-backfill-prio/data 2020-03-11T22:48:41.881 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:41.882 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj47-p1 td/osd-backfill-prio/data 2020-03-11T22:48:42.373 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:42.373 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj48-p1 td/osd-backfill-prio/data 2020-03-11T22:48:42.855 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:42.856 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj49-p1 td/osd-backfill-prio/data 2020-03-11T22:48:43.349 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:43.349 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj50-p1 td/osd-backfill-prio/data 2020-03-11T22:48:43.758 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:144: TEST_backfill_priority: expr 1 + 1 2020-03-11T22:48:43.759 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:144: TEST_backfill_priority: p=2 2020-03-11T22:48:43.759 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:138: TEST_backfill_priority: for pname in $pool1 $pool2 $pool3 2020-03-11T22:48:43.760 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: seq 1 50 2020-03-11T22:48:43.760 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:43.760 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj1-p2 td/osd-backfill-prio/data 2020-03-11T22:48:44.017 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:44.017 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj2-p2 td/osd-backfill-prio/data 2020-03-11T22:48:44.275 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:44.276 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj3-p2 td/osd-backfill-prio/data 2020-03-11T22:48:44.568 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:44.568 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj4-p2 td/osd-backfill-prio/data 2020-03-11T22:48:44.818 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:44.818 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj5-p2 td/osd-backfill-prio/data 2020-03-11T22:48:45.110 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:45.110 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj6-p2 td/osd-backfill-prio/data 2020-03-11T22:48:45.393 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:45.394 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj7-p2 td/osd-backfill-prio/data 2020-03-11T22:48:45.677 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:45.677 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj8-p2 td/osd-backfill-prio/data 2020-03-11T22:48:45.918 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:45.918 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj9-p2 td/osd-backfill-prio/data 2020-03-11T22:48:46.194 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:46.195 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj10-p2 td/osd-backfill-prio/data 2020-03-11T22:48:46.410 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:46.410 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj11-p2 td/osd-backfill-prio/data 2020-03-11T22:48:46.689 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:46.690 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj12-p2 td/osd-backfill-prio/data 2020-03-11T22:48:46.943 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:46.944 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj13-p2 td/osd-backfill-prio/data 2020-03-11T22:48:47.224 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:47.224 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj14-p2 td/osd-backfill-prio/data 2020-03-11T22:48:47.585 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:47.586 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj15-p2 td/osd-backfill-prio/data 2020-03-11T22:48:47.886 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:47.886 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj16-p2 td/osd-backfill-prio/data 2020-03-11T22:48:48.219 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:48.220 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj17-p2 td/osd-backfill-prio/data 2020-03-11T22:48:48.444 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:48.444 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj18-p2 td/osd-backfill-prio/data 2020-03-11T22:48:48.715 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:48.716 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj19-p2 td/osd-backfill-prio/data 2020-03-11T22:48:48.983 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:48.983 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj20-p2 td/osd-backfill-prio/data 2020-03-11T22:48:49.266 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:49.266 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj21-p2 td/osd-backfill-prio/data 2020-03-11T22:48:49.558 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:49.558 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj22-p2 td/osd-backfill-prio/data 2020-03-11T22:48:49.830 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:49.830 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj23-p2 td/osd-backfill-prio/data 2020-03-11T22:48:50.075 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:50.075 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj24-p2 td/osd-backfill-prio/data 2020-03-11T22:48:50.365 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:50.365 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj25-p2 td/osd-backfill-prio/data 2020-03-11T22:48:50.625 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:50.625 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj26-p2 td/osd-backfill-prio/data 2020-03-11T22:48:50.875 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:50.875 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj27-p2 td/osd-backfill-prio/data 2020-03-11T22:48:51.167 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:51.167 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj28-p2 td/osd-backfill-prio/data 2020-03-11T22:48:51.450 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:51.450 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj29-p2 td/osd-backfill-prio/data 2020-03-11T22:48:51.742 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:51.742 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj30-p2 td/osd-backfill-prio/data 2020-03-11T22:48:51.984 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:51.984 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj31-p2 td/osd-backfill-prio/data 2020-03-11T22:48:52.300 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:52.300 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj32-p2 td/osd-backfill-prio/data 2020-03-11T22:48:52.534 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:52.534 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj33-p2 td/osd-backfill-prio/data 2020-03-11T22:48:52.811 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:52.812 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj34-p2 td/osd-backfill-prio/data 2020-03-11T22:48:53.151 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:53.151 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj35-p2 td/osd-backfill-prio/data 2020-03-11T22:48:53.480 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:53.480 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj36-p2 td/osd-backfill-prio/data 2020-03-11T22:48:53.743 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:53.743 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj37-p2 td/osd-backfill-prio/data 2020-03-11T22:48:54.031 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:54.032 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj38-p2 td/osd-backfill-prio/data 2020-03-11T22:48:54.293 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:54.294 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj39-p2 td/osd-backfill-prio/data 2020-03-11T22:48:54.593 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:54.594 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj40-p2 td/osd-backfill-prio/data 2020-03-11T22:48:54.844 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:54.844 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj41-p2 td/osd-backfill-prio/data 2020-03-11T22:48:55.142 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:55.142 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj42-p2 td/osd-backfill-prio/data 2020-03-11T22:48:55.452 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:55.452 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj43-p2 td/osd-backfill-prio/data 2020-03-11T22:48:55.760 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:55.760 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj44-p2 td/osd-backfill-prio/data 2020-03-11T22:48:56.019 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:56.019 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj45-p2 td/osd-backfill-prio/data 2020-03-11T22:48:56.277 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:56.278 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj46-p2 td/osd-backfill-prio/data 2020-03-11T22:48:56.586 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:56.586 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj47-p2 td/osd-backfill-prio/data 2020-03-11T22:48:56.828 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:56.828 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj48-p2 td/osd-backfill-prio/data 2020-03-11T22:48:57.111 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:57.111 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj49-p2 td/osd-backfill-prio/data 2020-03-11T22:48:57.403 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:57.403 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj50-p2 td/osd-backfill-prio/data 2020-03-11T22:48:57.687 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:144: TEST_backfill_priority: expr 2 + 1 2020-03-11T22:48:57.687 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:144: TEST_backfill_priority: p=3 2020-03-11T22:48:57.688 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:138: TEST_backfill_priority: for pname in $pool1 $pool2 $pool3 2020-03-11T22:48:57.688 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: seq 1 50 2020-03-11T22:48:57.689 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:57.689 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj1-p3 td/osd-backfill-prio/data 2020-03-11T22:48:58.000 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:58.001 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj2-p3 td/osd-backfill-prio/data 2020-03-11T22:48:58.293 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:58.293 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj3-p3 td/osd-backfill-prio/data 2020-03-11T22:48:58.554 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:58.554 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj4-p3 td/osd-backfill-prio/data 2020-03-11T22:48:58.867 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:58.867 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj5-p3 td/osd-backfill-prio/data 2020-03-11T22:48:59.128 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:59.129 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj6-p3 td/osd-backfill-prio/data 2020-03-11T22:48:59.486 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:59.486 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj7-p3 td/osd-backfill-prio/data 2020-03-11T22:48:59.754 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:59.754 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj8-p3 td/osd-backfill-prio/data 2020-03-11T22:49:00.012 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:00.012 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj9-p3 td/osd-backfill-prio/data 2020-03-11T22:49:00.312 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:00.313 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj10-p3 td/osd-backfill-prio/data 2020-03-11T22:49:00.520 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:00.521 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj11-p3 td/osd-backfill-prio/data 2020-03-11T22:49:00.804 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:00.804 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj12-p3 td/osd-backfill-prio/data 2020-03-11T22:49:01.029 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:01.029 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj13-p3 td/osd-backfill-prio/data 2020-03-11T22:49:01.348 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:01.348 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj14-p3 td/osd-backfill-prio/data 2020-03-11T22:49:01.613 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:01.613 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj15-p3 td/osd-backfill-prio/data 2020-03-11T22:49:01.883 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:01.883 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj16-p3 td/osd-backfill-prio/data 2020-03-11T22:49:02.129 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:02.130 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj17-p3 td/osd-backfill-prio/data 2020-03-11T22:49:02.426 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:02.426 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj18-p3 td/osd-backfill-prio/data 2020-03-11T22:49:02.688 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:02.689 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj19-p3 td/osd-backfill-prio/data 2020-03-11T22:49:03.002 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:03.002 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj20-p3 td/osd-backfill-prio/data 2020-03-11T22:49:03.497 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:03.497 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj21-p3 td/osd-backfill-prio/data 2020-03-11T22:49:03.738 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:03.739 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj22-p3 td/osd-backfill-prio/data 2020-03-11T22:49:04.039 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:04.039 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj23-p3 td/osd-backfill-prio/data 2020-03-11T22:49:04.289 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:04.289 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj24-p3 td/osd-backfill-prio/data 2020-03-11T22:49:04.589 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:04.589 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj25-p3 td/osd-backfill-prio/data 2020-03-11T22:49:04.839 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:04.840 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj26-p3 td/osd-backfill-prio/data 2020-03-11T22:49:05.131 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:05.132 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj27-p3 td/osd-backfill-prio/data 2020-03-11T22:49:05.431 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:05.431 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj28-p3 td/osd-backfill-prio/data 2020-03-11T22:49:05.724 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:05.725 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj29-p3 td/osd-backfill-prio/data 2020-03-11T22:49:05.981 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:05.981 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj30-p3 td/osd-backfill-prio/data 2020-03-11T22:49:06.284 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:06.284 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj31-p3 td/osd-backfill-prio/data 2020-03-11T22:49:06.540 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:06.540 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj32-p3 td/osd-backfill-prio/data 2020-03-11T22:49:06.781 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:06.782 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj33-p3 td/osd-backfill-prio/data 2020-03-11T22:49:07.073 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:07.074 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj34-p3 td/osd-backfill-prio/data 2020-03-11T22:49:07.378 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:07.378 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj35-p3 td/osd-backfill-prio/data 2020-03-11T22:49:07.698 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:07.699 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj36-p3 td/osd-backfill-prio/data 2020-03-11T22:49:08.013 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:08.013 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj37-p3 td/osd-backfill-prio/data 2020-03-11T22:49:08.315 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:08.316 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj38-p3 td/osd-backfill-prio/data 2020-03-11T22:49:08.557 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:08.557 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj39-p3 td/osd-backfill-prio/data 2020-03-11T22:49:08.933 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:08.933 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj40-p3 td/osd-backfill-prio/data 2020-03-11T22:49:09.174 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:09.174 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj41-p3 td/osd-backfill-prio/data 2020-03-11T22:49:09.516 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:09.516 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj42-p3 td/osd-backfill-prio/data 2020-03-11T22:49:09.783 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:09.783 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj43-p3 td/osd-backfill-prio/data 2020-03-11T22:49:10.083 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:10.083 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj44-p3 td/osd-backfill-prio/data 2020-03-11T22:49:10.333 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:10.333 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj45-p3 td/osd-backfill-prio/data 2020-03-11T22:49:10.619 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:10.619 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj46-p3 td/osd-backfill-prio/data 2020-03-11T22:49:10.875 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:10.875 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj47-p3 td/osd-backfill-prio/data 2020-03-11T22:49:11.147 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:11.147 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj48-p3 td/osd-backfill-prio/data 2020-03-11T22:49:11.392 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:11.392 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj49-p3 td/osd-backfill-prio/data 2020-03-11T22:49:11.600 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:49:11.600 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj50-p3 td/osd-backfill-prio/data 2020-03-11T22:49:11.909 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:144: TEST_backfill_priority: expr 3 + 1 2020-03-11T22:49:11.910 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:144: TEST_backfill_priority: p=4 2020-03-11T22:49:11.911 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:147: TEST_backfill_priority: get_not_primary test1 obj1-p1 2020-03-11T22:49:11.911 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1166: get_not_primary: local poolname=test1 2020-03-11T22:49:11.911 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1167: get_not_primary: local objectname=obj1-p1 2020-03-11T22:49:11.911 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1169: get_not_primary: get_primary test1 obj1-p1 2020-03-11T22:49:11.911 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1133: get_primary: local poolname=test1 2020-03-11T22:49:11.911 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local objectname=obj1-p1 2020-03-11T22:49:11.912 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1136: get_primary: ceph --format json osd map test1 obj1-p1 2020-03-11T22:49:11.912 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: jq .acting_primary 2020-03-11T22:49:12.292 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1169: get_not_primary: local primary=1 2020-03-11T22:49:12.292 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: ceph --format json osd map test1 obj1-p1 2020-03-11T22:49:12.292 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2020-03-11T22:49:12.666 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:147: TEST_backfill_priority: local otherosd=0 2020-03-11T22:49:12.666 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:149: TEST_backfill_priority: ceph pg dump pgs 2020-03-11T22:49:13.016 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:49:13.016 INFO:tasks.workunit.client.0.smithi059.stdout:7.0 46 0 0 0 0 480247808 0 0 37 37 active+clean 2020-03-11T22:48:14.500920+0000 97'137 97:184 [1] 1 [1] 1 0'0 2020-03-11T22:47:44.646052+0000 0'0 2020-03-11T22:47:44.646052+0000 0 2020-03-11T22:49:13.017 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2020-03-11T22:47:33.257226+0000 97'150 97:219 [1,0] 1 [1,0] 1 0'0 2020-03-11T22:47:30.333509+0000 0'0 2020-03-11T22:47:30.333509+0000 0 2020-03-11T22:49:13.017 INFO:tasks.workunit.client.0.smithi059.stdout:3.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2020-03-11T22:48:12.753837+0000 97'150 97:219 [1] 1 [1] 1 0'0 2020-03-11T22:47:35.075766+0000 0'0 2020-03-11T22:47:35.075766+0000 0 2020-03-11T22:49:13.017 INFO:tasks.workunit.client.0.smithi059.stdout: 2020-03-11T22:49:13.017 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:49:13.018 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2020-03-11T22:49:13.026 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:150: TEST_backfill_priority: ERRORS=0 2020-03-11T22:49:13.026 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:152: TEST_backfill_priority: ceph osd set nobackfill 2020-03-11T22:49:13.640 INFO:tasks.workunit.client.0.smithi059.stderr:nobackfill is set 2020-03-11T22:49:13.648 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:153: TEST_backfill_priority: ceph osd set noout 2020-03-11T22:49:14.354 INFO:tasks.workunit.client.0.smithi059.stderr:noout is set 2020-03-11T22:49:14.362 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:157: TEST_backfill_priority: ceph osd pool set test7 size 2 2020-03-11T22:49:15.273 INFO:tasks.workunit.client.0.smithi059.stderr:set pool 7 size to 2 2020-03-11T22:49:15.280 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:158: TEST_backfill_priority: sleep 2 2020-03-11T22:49:17.282 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:160: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:49:17.282 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:49:17.283 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:49:17.283 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:49:17.283 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:49:17.283 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56462 2020-03-11T22:49:17.283 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56462/ceph-osd.1.asok 2020-03-11T22:49:17.284 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:160: TEST_backfill_priority: CEPH_ARGS= 2020-03-11T22:49:17.284 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:160: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:49:17.386 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2020-03-11T22:49:17.386 INFO:tasks.workunit.client.0.smithi059.stdout: "local_reservations": { 2020-03-11T22:49:17.386 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:49:17.387 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:49:17.387 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [], 2020-03-11T22:49:17.387 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [ 2020-03-11T22:49:17.387 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:49:17.387 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "7.0", 2020-03-11T22:49:17.387 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 151, 2020-03-11T22:49:17.387 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:49:17.388 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:49:17.388 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:49:17.388 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:49:17.388 INFO:tasks.workunit.client.0.smithi059.stdout: "remote_reservations": { 2020-03-11T22:49:17.388 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:49:17.388 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:49:17.388 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [], 2020-03-11T22:49:17.388 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [] 2020-03-11T22:49:17.389 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:49:17.389 INFO:tasks.workunit.client.0.smithi059.stdout:} 2020-03-11T22:49:17.394 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:163: TEST_backfill_priority: seq 1 10 2020-03-11T22:49:17.395 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:163: TEST_backfill_priority: for i in $(seq 1 $max_tries) 2020-03-11T22:49:17.395 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:165: TEST_backfill_priority: ceph pg force-backfill 7.0 2020-03-11T22:49:17.395 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:165: TEST_backfill_priority: grep -q 'doesn'\''t require backfilling' 2020-03-11T22:49:17.752 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:166: TEST_backfill_priority: break 2020-03-11T22:49:17.752 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:174: TEST_backfill_priority: flush_pg_stats 2020-03-11T22:49:17.752 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:49:17.752 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:49:18.124 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:49:18.124 INFO:tasks.workunit.client.0.smithi059.stderr:1 2020-03-11T22:49:18.124 INFO:tasks.workunit.client.0.smithi059.stderr:2 2020-03-11T22:49:18.125 INFO:tasks.workunit.client.0.smithi059.stderr:3 2020-03-11T22:49:18.125 INFO:tasks.workunit.client.0.smithi059.stderr:4' 2020-03-11T22:49:18.125 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:49:18.125 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:49:18.125 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:49:18.297 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836513 2020-03-11T22:49:18.297 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836513' 2020-03-11T22:49:18.297 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:49:18.297 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:49:18.430 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672992 2020-03-11T22:49:18.430 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836513 1-42949672992' 2020-03-11T22:49:18.430 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:49:18.430 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:49:18.562 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509472 2020-03-11T22:49:18.562 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836513 1-42949672992 2-64424509472' 2020-03-11T22:49:18.563 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:49:18.563 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:49:18.695 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=85899345949 2020-03-11T22:49:18.695 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836513 1-42949672992 2-64424509472 3-85899345949' 2020-03-11T22:49:18.695 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:49:18.696 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:49:18.827 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=107374182428 2020-03-11T22:49:18.828 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836513 1-42949672992 2-64424509472 3-85899345949 4-107374182428' 2020-03-11T22:49:18.828 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:49:18.828 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836513 2020-03-11T22:49:18.828 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:49:18.830 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:49:18.831 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836513 2020-03-11T22:49:18.831 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:49:18.831 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836513 2020-03-11T22:49:18.831 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836513' 2020-03-11T22:49:18.832 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:49:18.832 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.0 seq 21474836513 2020-03-11T22:49:19.196 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836511 -lt 21474836513 2020-03-11T22:49:19.196 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:49:20.197 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:49:20.198 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:49:20.577 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836513 -lt 21474836513 2020-03-11T22:49:20.578 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:49:20.578 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672992 2020-03-11T22:49:20.578 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:49:20.579 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:49:20.579 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672992 2020-03-11T22:49:20.579 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:49:20.580 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672992 2020-03-11T22:49:20.580 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672992' 2020-03-11T22:49:20.580 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:49:20.581 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.1 seq 42949672992 2020-03-11T22:49:20.971 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672992 -lt 42949672992 2020-03-11T22:49:20.971 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:49:20.971 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509472 2020-03-11T22:49:20.971 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:49:20.972 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:49:20.972 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509472 2020-03-11T22:49:20.973 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:49:20.973 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509472 2020-03-11T22:49:20.974 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509472' 2020-03-11T22:49:20.974 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.2 seq 64424509472 2020-03-11T22:49:20.974 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:49:21.376 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509472 -lt 64424509472 2020-03-11T22:49:21.377 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:49:21.377 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-85899345949 2020-03-11T22:49:21.377 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:49:21.378 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:49:21.378 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-85899345949 2020-03-11T22:49:21.379 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:49:21.379 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=85899345949 2020-03-11T22:49:21.379 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 85899345949' 2020-03-11T22:49:21.380 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:49:21.380 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.3 seq 85899345949 2020-03-11T22:49:21.749 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 85899345950 -lt 85899345949 2020-03-11T22:49:21.750 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:49:21.750 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-107374182428 2020-03-11T22:49:21.750 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:49:21.751 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:49:21.751 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-107374182428 2020-03-11T22:49:21.751 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:49:21.752 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=107374182428 2020-03-11T22:49:21.752 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 107374182428' 2020-03-11T22:49:21.753 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:49:21.753 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.4 seq 107374182428 2020-03-11T22:49:22.125 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 107374182429 -lt 107374182428 2020-03-11T22:49:22.127 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:175: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:49:22.127 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:49:22.127 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:49:22.127 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:49:22.127 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:49:22.127 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56462 2020-03-11T22:49:22.127 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56462/ceph-osd.1.asok 2020-03-11T22:49:22.128 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:175: TEST_backfill_priority: CEPH_ARGS= 2020-03-11T22:49:22.128 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:175: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:49:22.219 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2020-03-11T22:49:22.219 INFO:tasks.workunit.client.0.smithi059.stdout: "local_reservations": { 2020-03-11T22:49:22.219 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:49:22.219 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:49:22.219 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [], 2020-03-11T22:49:22.220 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [ 2020-03-11T22:49:22.220 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:49:22.220 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "7.0", 2020-03-11T22:49:22.220 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 254, 2020-03-11T22:49:22.220 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:49:22.220 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:49:22.220 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:49:22.221 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:49:22.221 INFO:tasks.workunit.client.0.smithi059.stdout: "remote_reservations": { 2020-03-11T22:49:22.221 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:49:22.221 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:49:22.221 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [], 2020-03-11T22:49:22.221 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [] 2020-03-11T22:49:22.221 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:49:22.221 INFO:tasks.workunit.client.0.smithi059.stdout:} 2020-03-11T22:49:22.226 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:177: TEST_backfill_priority: ceph osd out osd.0 2020-03-11T22:49:22.828 INFO:tasks.workunit.client.0.smithi059.stderr:osd.0 is already out. 2020-03-11T22:49:22.836 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:178: TEST_backfill_priority: sleep 2 2020-03-11T22:49:24.837 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:179: TEST_backfill_priority: flush_pg_stats 2020-03-11T22:49:24.838 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:49:24.838 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:49:25.220 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:49:25.220 INFO:tasks.workunit.client.0.smithi059.stderr:1 2020-03-11T22:49:25.220 INFO:tasks.workunit.client.0.smithi059.stderr:2 2020-03-11T22:49:25.220 INFO:tasks.workunit.client.0.smithi059.stderr:3 2020-03-11T22:49:25.220 INFO:tasks.workunit.client.0.smithi059.stderr:4' 2020-03-11T22:49:25.220 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:49:25.221 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:49:25.221 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:49:25.371 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836517 2020-03-11T22:49:25.371 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836517' 2020-03-11T22:49:25.371 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:49:25.371 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:49:25.503 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672995 2020-03-11T22:49:25.503 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836517 1-42949672995' 2020-03-11T22:49:25.503 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:49:25.503 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:49:25.635 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509475 2020-03-11T22:49:25.635 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836517 1-42949672995 2-64424509475' 2020-03-11T22:49:25.636 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:49:25.636 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:49:25.768 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=85899345953 2020-03-11T22:49:25.768 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836517 1-42949672995 2-64424509475 3-85899345953' 2020-03-11T22:49:25.768 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:49:25.768 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:49:25.901 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=107374182432 2020-03-11T22:49:25.901 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836517 1-42949672995 2-64424509475 3-85899345953 4-107374182432' 2020-03-11T22:49:25.901 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:49:25.902 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836517 2020-03-11T22:49:25.902 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:49:25.902 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:49:25.903 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836517 2020-03-11T22:49:25.903 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:49:25.903 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836517 2020-03-11T22:49:25.904 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836517' 2020-03-11T22:49:25.904 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:49:25.904 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.0 seq 21474836517 2020-03-11T22:49:26.272 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836515 -lt 21474836517 2020-03-11T22:49:26.272 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:49:27.274 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:49:27.274 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:49:27.655 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836517 -lt 21474836517 2020-03-11T22:49:27.655 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:49:27.656 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672995 2020-03-11T22:49:27.656 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:49:27.657 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:49:27.658 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672995 2020-03-11T22:49:27.658 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:49:27.658 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672995 2020-03-11T22:49:27.659 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672995' 2020-03-11T22:49:27.659 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:49:27.659 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.1 seq 42949672995 2020-03-11T22:49:28.049 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672996 -lt 42949672995 2020-03-11T22:49:28.049 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:49:28.050 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509475 2020-03-11T22:49:28.050 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:49:28.051 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:49:28.051 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509475 2020-03-11T22:49:28.051 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:49:28.052 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509475 2020-03-11T22:49:28.052 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509475' 2020-03-11T22:49:28.052 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:49:28.053 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.2 seq 64424509475 2020-03-11T22:49:28.418 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509475 -lt 64424509475 2020-03-11T22:49:28.418 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:49:28.419 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-85899345953 2020-03-11T22:49:28.419 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:49:28.419 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:49:28.420 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-85899345953 2020-03-11T22:49:28.420 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:49:28.421 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=85899345953 2020-03-11T22:49:28.421 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 85899345953' 2020-03-11T22:49:28.421 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:49:28.421 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.3 seq 85899345953 2020-03-11T22:49:28.790 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 85899345953 -lt 85899345953 2020-03-11T22:49:28.790 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:49:28.790 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-107374182432 2020-03-11T22:49:28.791 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:49:28.791 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:49:28.791 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-107374182432 2020-03-11T22:49:28.792 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:49:28.792 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=107374182432 2020-03-11T22:49:28.793 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 107374182432' 2020-03-11T22:49:28.793 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:49:28.793 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.4 seq 107374182432 2020-03-11T22:49:29.161 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 107374182432 -lt 107374182432 2020-03-11T22:49:29.163 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:180: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:49:29.163 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:49:29.163 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:49:29.163 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:49:29.163 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:49:29.163 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56462 2020-03-11T22:49:29.163 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56462/ceph-osd.1.asok 2020-03-11T22:49:29.164 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:180: TEST_backfill_priority: CEPH_ARGS= 2020-03-11T22:49:29.164 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:180: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:49:29.255 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2020-03-11T22:49:29.255 INFO:tasks.workunit.client.0.smithi059.stdout: "local_reservations": { 2020-03-11T22:49:29.255 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:49:29.255 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:49:29.255 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [ 2020-03-11T22:49:29.256 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:49:29.256 INFO:tasks.workunit.client.0.smithi059.stdout: "priority": 110, 2020-03-11T22:49:29.256 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2020-03-11T22:49:29.256 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:49:29.256 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "1.0", 2020-03-11T22:49:29.256 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 110, 2020-03-11T22:49:29.256 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:49:29.257 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:49:29.257 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:49:29.257 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:49:29.257 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:49:29.257 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [ 2020-03-11T22:49:29.257 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:49:29.257 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "7.0", 2020-03-11T22:49:29.258 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 254, 2020-03-11T22:49:29.258 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:49:29.258 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:49:29.258 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:49:29.258 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:49:29.258 INFO:tasks.workunit.client.0.smithi059.stdout: "remote_reservations": { 2020-03-11T22:49:29.258 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:49:29.259 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:49:29.259 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [], 2020-03-11T22:49:29.259 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [] 2020-03-11T22:49:29.259 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:49:29.259 INFO:tasks.workunit.client.0.smithi059.stdout:} 2020-03-11T22:49:29.263 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:181: TEST_backfill_priority: ceph pg dump pgs 2020-03-11T22:49:29.607 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:49:29.607 INFO:tasks.workunit.client.0.smithi059.stdout:7.0 50 0 50 0 0 524288000 0 0 50 50 active+undersized+degraded+remapped+backfilling+forced_backfill 2020-03-11T22:49:17.643944+0000 97'150 107:217 [1,4] 1 [1] 1 0'0 2020-03-11T22:47:44.646052+0000 0'0 2020-03-11T22:47:44.646052+0000 0 2020-03-11T22:49:29.608 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 50 0 0 50 0 524288000 0 0 50 50 active+remapped+backfill_wait 2020-03-11T22:49:23.728030+0000 97'150 108:237 [1,2] 1 [1,0] 1 0'0 2020-03-11T22:47:30.333509+0000 0'0 2020-03-11T22:47:30.333509+0000 0 2020-03-11T22:49:29.608 INFO:tasks.workunit.client.0.smithi059.stdout:3.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2020-03-11T22:48:12.753837+0000 97'150 107:229 [1] 1 [1] 1 0'0 2020-03-11T22:47:35.075766+0000 0'0 2020-03-11T22:47:35.075766+0000 0 2020-03-11T22:49:29.608 INFO:tasks.workunit.client.0.smithi059.stdout: 2020-03-11T22:49:29.608 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:49:29.609 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2020-03-11T22:49:29.617 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:183: TEST_backfill_priority: ceph osd pool set test3 size 2 2020-03-11T22:49:30.235 INFO:tasks.workunit.client.0.smithi059.stderr:set pool 3 size to 2 2020-03-11T22:49:30.242 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:184: TEST_backfill_priority: sleep 2 2020-03-11T22:49:32.244 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:185: TEST_backfill_priority: flush_pg_stats 2020-03-11T22:49:32.244 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:49:32.245 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:49:32.634 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:49:32.634 INFO:tasks.workunit.client.0.smithi059.stderr:1 2020-03-11T22:49:32.634 INFO:tasks.workunit.client.0.smithi059.stderr:2 2020-03-11T22:49:32.634 INFO:tasks.workunit.client.0.smithi059.stderr:3 2020-03-11T22:49:32.634 INFO:tasks.workunit.client.0.smithi059.stderr:4' 2020-03-11T22:49:32.635 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:49:32.635 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:49:32.635 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:49:32.800 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836520 2020-03-11T22:49:32.800 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836520' 2020-03-11T22:49:32.800 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:49:32.800 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:49:32.927 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672999 2020-03-11T22:49:32.927 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836520 1-42949672999' 2020-03-11T22:49:32.927 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:49:32.927 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:49:33.059 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509479 2020-03-11T22:49:33.059 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836520 1-42949672999 2-64424509479' 2020-03-11T22:49:33.059 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:49:33.059 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:49:33.191 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=85899345956 2020-03-11T22:49:33.191 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836520 1-42949672999 2-64424509479 3-85899345956' 2020-03-11T22:49:33.191 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:49:33.192 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:49:33.323 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=107374182435 2020-03-11T22:49:33.323 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836520 1-42949672999 2-64424509479 3-85899345956 4-107374182435' 2020-03-11T22:49:33.324 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:49:33.324 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836520 2020-03-11T22:49:33.324 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:49:33.325 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:49:33.325 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836520 2020-03-11T22:49:33.325 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:49:33.326 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836520 2020-03-11T22:49:33.326 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836520' 2020-03-11T22:49:33.326 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:49:33.327 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.0 seq 21474836520 2020-03-11T22:49:33.692 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836520 -lt 21474836520 2020-03-11T22:49:33.692 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:49:33.693 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672999 2020-03-11T22:49:33.693 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:49:33.693 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:49:33.694 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672999 2020-03-11T22:49:33.694 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:49:33.695 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672999 2020-03-11T22:49:33.695 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672999' 2020-03-11T22:49:33.695 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.1 seq 42949672999 2020-03-11T22:49:33.695 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:49:34.108 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672999 -lt 42949672999 2020-03-11T22:49:34.108 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:49:34.109 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509479 2020-03-11T22:49:34.109 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:49:34.109 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:49:34.110 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509479 2020-03-11T22:49:34.110 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:49:34.111 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509479 2020-03-11T22:49:34.111 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509479' 2020-03-11T22:49:34.112 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:49:34.112 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.2 seq 64424509479 2020-03-11T22:49:34.481 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509479 -lt 64424509479 2020-03-11T22:49:34.481 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:49:34.481 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-85899345956 2020-03-11T22:49:34.482 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:49:34.482 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:49:34.483 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-85899345956 2020-03-11T22:49:34.483 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:49:34.483 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=85899345956 2020-03-11T22:49:34.484 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 85899345956' 2020-03-11T22:49:34.484 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:49:34.484 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.3 seq 85899345956 2020-03-11T22:49:34.850 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 85899345956 -lt 85899345956 2020-03-11T22:49:34.851 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:49:34.851 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-107374182435 2020-03-11T22:49:34.851 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:49:34.852 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:49:34.852 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-107374182435 2020-03-11T22:49:34.852 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:49:34.853 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=107374182435 2020-03-11T22:49:34.854 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 107374182435' 2020-03-11T22:49:34.854 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:49:34.854 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.4 seq 107374182435 2020-03-11T22:49:35.226 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 107374182433 -lt 107374182435 2020-03-11T22:49:35.226 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:49:36.227 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:49:36.227 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:49:36.617 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 107374182435 -lt 107374182435 2020-03-11T22:49:36.618 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:186: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:49:36.618 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:49:36.618 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:49:36.619 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:49:36.619 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:49:36.619 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56462 2020-03-11T22:49:36.619 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56462/ceph-osd.1.asok 2020-03-11T22:49:36.619 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:186: TEST_backfill_priority: CEPH_ARGS= 2020-03-11T22:49:36.619 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:186: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:49:36.718 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:187: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:49:36.719 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2020-03-11T22:49:36.719 INFO:tasks.workunit.client.0.smithi059.stdout: "local_reservations": { 2020-03-11T22:49:36.719 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:49:36.720 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:49:36.720 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [ 2020-03-11T22:49:36.720 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:49:36.720 INFO:tasks.workunit.client.0.smithi059.stdout: "priority": 110, 2020-03-11T22:49:36.721 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2020-03-11T22:49:36.721 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:49:36.721 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "1.0", 2020-03-11T22:49:36.722 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 110, 2020-03-11T22:49:36.722 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:49:36.722 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:49:36.723 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:49:36.723 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:49:36.723 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:49:36.723 INFO:tasks.workunit.client.0.smithi059.stdout: "priority": 151, 2020-03-11T22:49:36.724 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2020-03-11T22:49:36.724 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:49:36.724 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "3.0", 2020-03-11T22:49:36.724 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 151, 2020-03-11T22:49:36.725 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:49:36.725 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:49:36.725 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:49:36.725 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:49:36.726 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:49:36.726 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [ 2020-03-11T22:49:36.726 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:49:36.727 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "7.0", 2020-03-11T22:49:36.727 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 254, 2020-03-11T22:49:36.727 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:49:36.728 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:49:36.728 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:49:36.728 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:49:36.728 INFO:tasks.workunit.client.0.smithi059.stdout: "remote_reservations": { 2020-03-11T22:49:36.729 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:49:36.729 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:49:36.729 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [], 2020-03-11T22:49:36.729 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [] 2020-03-11T22:49:36.730 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:49:36.730 INFO:tasks.workunit.client.0.smithi059.stdout:} 2020-03-11T22:49:36.730 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:188: TEST_backfill_priority: ceph pg dump pgs 2020-03-11T22:49:37.061 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:49:37.061 INFO:tasks.workunit.client.0.smithi059.stdout:7.0 50 0 50 0 0 524288000 0 0 50 50 active+undersized+degraded+remapped+backfilling+forced_backfill 2020-03-11T22:49:17.643944+0000 97'150 111:220 [1,4] 1 [1] 1 0'0 2020-03-11T22:47:44.646052+0000 0'0 2020-03-11T22:47:44.646052+0000 0 2020-03-11T22:49:37.061 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 50 0 0 50 0 524288000 0 0 50 50 active+remapped+backfill_wait 2020-03-11T22:49:23.728030+0000 97'150 111:240 [1,2] 1 [1,0] 1 0'0 2020-03-11T22:47:30.333509+0000 0'0 2020-03-11T22:47:30.333509+0000 0 2020-03-11T22:49:37.062 INFO:tasks.workunit.client.0.smithi059.stdout:3.0 50 0 50 0 0 524288000 0 0 50 50 active+undersized+degraded+remapped+backfill_wait 2020-03-11T22:49:30.924789+0000 97'150 112:239 [1,2] 1 [1] 1 0'0 2020-03-11T22:47:35.075766+0000 0'0 2020-03-11T22:47:35.075766+0000 0 2020-03-11T22:49:37.062 INFO:tasks.workunit.client.0.smithi059.stdout: 2020-03-11T22:49:37.062 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:49:37.063 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2020-03-11T22:49:37.071 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:190: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:49:37.072 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:190: TEST_backfill_priority: jq '(.local_reservations.queues[].items[] | select(.item == "1.0")).prio' 2020-03-11T22:49:37.074 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:190: TEST_backfill_priority: PRIO=110 2020-03-11T22:49:37.074 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:191: TEST_backfill_priority: '[' 110 '!=' 110 ']' 2020-03-11T22:49:37.075 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:198: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:49:37.075 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:198: TEST_backfill_priority: jq '.local_reservations.in_progress[0].item' 2020-03-11T22:49:37.077 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:198: TEST_backfill_priority: eval 'ITEM="7.0"' 2020-03-11T22:49:37.077 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:198: TEST_backfill_priority: ITEM=7.0 2020-03-11T22:49:37.077 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:199: TEST_backfill_priority: '[' 7.0 '!=' 7.0 ']' 2020-03-11T22:49:37.078 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:204: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:49:37.078 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:204: TEST_backfill_priority: jq '.local_reservations.in_progress[0].prio' 2020-03-11T22:49:37.079 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:204: TEST_backfill_priority: PRIO=254 2020-03-11T22:49:37.080 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:205: TEST_backfill_priority: '[' 254 '!=' 254 ']' 2020-03-11T22:49:37.080 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:213: TEST_backfill_priority: seq 1 10 2020-03-11T22:49:37.080 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:213: TEST_backfill_priority: for i in $(seq 1 $max_tries) 2020-03-11T22:49:37.081 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:215: TEST_backfill_priority: ceph pg force-backfill 3.0 2020-03-11T22:49:37.081 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:215: TEST_backfill_priority: grep -q 'doesn'\''t require backfilling' 2020-03-11T22:49:37.440 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:216: TEST_backfill_priority: break 2020-03-11T22:49:37.440 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:224: TEST_backfill_priority: sleep 2 2020-03-11T22:49:39.445 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:225: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:49:39.445 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:49:39.445 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:49:39.445 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:49:39.446 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:49:39.446 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56462 2020-03-11T22:49:39.446 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56462/ceph-osd.1.asok 2020-03-11T22:49:39.446 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:225: TEST_backfill_priority: CEPH_ARGS= 2020-03-11T22:49:39.446 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:225: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:49:39.554 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:226: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:49:39.555 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2020-03-11T22:49:39.555 INFO:tasks.workunit.client.0.smithi059.stdout: "local_reservations": { 2020-03-11T22:49:39.555 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:49:39.555 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:49:39.555 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [ 2020-03-11T22:49:39.555 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:49:39.556 INFO:tasks.workunit.client.0.smithi059.stdout: "priority": 110, 2020-03-11T22:49:39.556 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2020-03-11T22:49:39.556 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:49:39.556 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "1.0", 2020-03-11T22:49:39.556 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 110, 2020-03-11T22:49:39.556 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:49:39.556 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:49:39.557 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:49:39.557 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:49:39.557 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:49:39.557 INFO:tasks.workunit.client.0.smithi059.stdout: "priority": 254, 2020-03-11T22:49:39.557 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2020-03-11T22:49:39.557 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:49:39.557 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "3.0", 2020-03-11T22:49:39.558 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 254, 2020-03-11T22:49:39.558 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:49:39.558 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:49:39.558 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:49:39.558 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:49:39.558 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:49:39.558 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [ 2020-03-11T22:49:39.559 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:49:39.559 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "7.0", 2020-03-11T22:49:39.559 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 254, 2020-03-11T22:49:39.559 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:49:39.559 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:49:39.559 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:49:39.559 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:49:39.560 INFO:tasks.workunit.client.0.smithi059.stdout: "remote_reservations": { 2020-03-11T22:49:39.560 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:49:39.560 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:49:39.560 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [], 2020-03-11T22:49:39.560 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [] 2020-03-11T22:49:39.560 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:49:39.560 INFO:tasks.workunit.client.0.smithi059.stdout:} 2020-03-11T22:49:39.561 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:227: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:49:39.562 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:227: TEST_backfill_priority: jq '(.local_reservations.queues[].items[] | select(.item == "3.0")).prio' 2020-03-11T22:49:39.562 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:227: TEST_backfill_priority: PRIO=254 2020-03-11T22:49:39.562 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:228: TEST_backfill_priority: '[' 254 '!=' 254 ']' 2020-03-11T22:49:39.562 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:233: TEST_backfill_priority: flush_pg_stats 2020-03-11T22:49:39.562 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:49:39.562 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:49:39.930 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:49:39.930 INFO:tasks.workunit.client.0.smithi059.stderr:1 2020-03-11T22:49:39.931 INFO:tasks.workunit.client.0.smithi059.stderr:2 2020-03-11T22:49:39.931 INFO:tasks.workunit.client.0.smithi059.stderr:3 2020-03-11T22:49:39.931 INFO:tasks.workunit.client.0.smithi059.stderr:4' 2020-03-11T22:49:39.931 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:49:39.931 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:49:39.931 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:49:40.100 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836524 2020-03-11T22:49:40.100 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836524' 2020-03-11T22:49:40.101 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:49:40.101 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:49:40.232 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949673002 2020-03-11T22:49:40.233 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836524 1-42949673002' 2020-03-11T22:49:40.233 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:49:40.233 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:49:40.365 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509482 2020-03-11T22:49:40.365 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836524 1-42949673002 2-64424509482' 2020-03-11T22:49:40.365 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:49:40.365 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:49:40.497 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=85899345960 2020-03-11T22:49:40.497 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836524 1-42949673002 2-64424509482 3-85899345960' 2020-03-11T22:49:40.497 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:49:40.497 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:49:40.630 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=107374182439 2020-03-11T22:49:40.630 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836524 1-42949673002 2-64424509482 3-85899345960 4-107374182439' 2020-03-11T22:49:40.630 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:49:40.631 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836524 2020-03-11T22:49:40.631 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:49:40.631 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:49:40.632 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836524 2020-03-11T22:49:40.632 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:49:40.633 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836524 2020-03-11T22:49:40.633 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836524' 2020-03-11T22:49:40.633 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:49:40.634 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.0 seq 21474836524 2020-03-11T22:49:41.001 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836521 -lt 21474836524 2020-03-11T22:49:41.001 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:49:42.002 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:49:42.003 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:49:42.382 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836524 -lt 21474836524 2020-03-11T22:49:42.382 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:49:42.383 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949673002 2020-03-11T22:49:42.383 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:49:42.383 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:49:42.384 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949673002 2020-03-11T22:49:42.384 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:49:42.385 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949673002 2020-03-11T22:49:42.385 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949673002' 2020-03-11T22:49:42.385 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:49:42.386 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.1 seq 42949673002 2020-03-11T22:49:42.826 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949673003 -lt 42949673002 2020-03-11T22:49:42.827 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:49:42.827 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509482 2020-03-11T22:49:42.827 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:49:42.828 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:49:42.828 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509482 2020-03-11T22:49:42.829 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:49:42.829 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509482 2020-03-11T22:49:42.830 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509482' 2020-03-11T22:49:42.830 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:49:42.830 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.2 seq 64424509482 2020-03-11T22:49:43.194 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509482 -lt 64424509482 2020-03-11T22:49:43.194 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:49:43.195 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-85899345960 2020-03-11T22:49:43.195 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:49:43.196 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:49:43.196 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-85899345960 2020-03-11T22:49:43.196 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:49:43.197 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=85899345960 2020-03-11T22:49:43.197 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 85899345960' 2020-03-11T22:49:43.198 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:49:43.198 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.3 seq 85899345960 2020-03-11T22:49:43.571 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 85899345960 -lt 85899345960 2020-03-11T22:49:43.572 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:49:43.572 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-107374182439 2020-03-11T22:49:43.573 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:49:43.573 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:49:43.573 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-107374182439 2020-03-11T22:49:43.574 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:49:43.574 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=107374182439 2020-03-11T22:49:43.574 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 107374182439' 2020-03-11T22:49:43.575 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.4 seq 107374182439 2020-03-11T22:49:43.575 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:49:43.949 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 107374182439 -lt 107374182439 2020-03-11T22:49:43.949 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:236: TEST_backfill_priority: ceph pg cancel-force-backfill 7.0 2020-03-11T22:49:44.302 INFO:tasks.workunit.client.0.smithi059.stderr:instructing pg(s) [7.0] on osd.1 to cancel-force-backfill; 2020-03-11T22:49:44.302 INFO:tasks.workunit.client.0.smithi059.stderr: 2020-03-11T22:49:44.310 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:237: TEST_backfill_priority: sleep 2 2020-03-11T22:49:46.312 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:238: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:49:46.312 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:49:46.312 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:49:46.313 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:49:46.313 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:49:46.313 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56462 2020-03-11T22:49:46.313 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56462/ceph-osd.1.asok 2020-03-11T22:49:46.313 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:238: TEST_backfill_priority: CEPH_ARGS= 2020-03-11T22:49:46.314 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:238: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:49:46.423 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:239: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:49:46.424 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2020-03-11T22:49:46.424 INFO:tasks.workunit.client.0.smithi059.stdout: "local_reservations": { 2020-03-11T22:49:46.424 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:49:46.424 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:49:46.424 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [ 2020-03-11T22:49:46.425 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:49:46.425 INFO:tasks.workunit.client.0.smithi059.stdout: "priority": 110, 2020-03-11T22:49:46.425 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2020-03-11T22:49:46.425 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:49:46.425 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "1.0", 2020-03-11T22:49:46.425 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 110, 2020-03-11T22:49:46.425 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:49:46.426 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:49:46.426 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:49:46.426 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:49:46.426 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:49:46.426 INFO:tasks.workunit.client.0.smithi059.stdout: "priority": 151, 2020-03-11T22:49:46.426 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2020-03-11T22:49:46.426 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:49:46.427 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "7.0", 2020-03-11T22:49:46.427 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 151, 2020-03-11T22:49:46.427 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:49:46.427 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:49:46.427 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:49:46.427 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:49:46.427 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:49:46.428 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [ 2020-03-11T22:49:46.428 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:49:46.428 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "3.0", 2020-03-11T22:49:46.428 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 254, 2020-03-11T22:49:46.428 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:49:46.428 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:49:46.428 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:49:46.428 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:49:46.429 INFO:tasks.workunit.client.0.smithi059.stdout: "remote_reservations": { 2020-03-11T22:49:46.429 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:49:46.429 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:49:46.429 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [], 2020-03-11T22:49:46.429 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [] 2020-03-11T22:49:46.429 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:49:46.429 INFO:tasks.workunit.client.0.smithi059.stdout:} 2020-03-11T22:49:46.431 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:240: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:49:46.431 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:240: TEST_backfill_priority: jq '(.local_reservations.queues[].items[] | select(.item == "7.0")).prio' 2020-03-11T22:49:46.431 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:240: TEST_backfill_priority: PRIO=151 2020-03-11T22:49:46.431 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:241: TEST_backfill_priority: '[' 151 '!=' 151 ']' 2020-03-11T22:49:46.431 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:247: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:49:46.431 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:247: TEST_backfill_priority: jq '.local_reservations.in_progress[0].item' 2020-03-11T22:49:46.432 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:247: TEST_backfill_priority: eval 'ITEM="3.0"' 2020-03-11T22:49:46.432 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:247: TEST_backfill_priority: ITEM=3.0 2020-03-11T22:49:46.432 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:248: TEST_backfill_priority: '[' 3.0 '!=' 3.0 ']' 2020-03-11T22:49:46.432 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:253: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:49:46.432 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:253: TEST_backfill_priority: jq '.local_reservations.in_progress[0].prio' 2020-03-11T22:49:46.433 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:253: TEST_backfill_priority: PRIO=254 2020-03-11T22:49:46.433 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:254: TEST_backfill_priority: '[' 254 '!=' 254 ']' 2020-03-11T22:49:46.434 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:261: TEST_backfill_priority: ceph pg cancel-force-backfill 3.0 2020-03-11T22:49:46.781 INFO:tasks.workunit.client.0.smithi059.stderr:instructing pg(s) [3.0] on osd.1 to cancel-force-backfill; 2020-03-11T22:49:46.781 INFO:tasks.workunit.client.0.smithi059.stderr: 2020-03-11T22:49:46.789 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:262: TEST_backfill_priority: sleep 5 2020-03-11T22:49:51.792 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:263: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:49:51.792 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:49:51.792 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:49:51.792 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:49:51.792 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:49:51.792 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56462 2020-03-11T22:49:51.793 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56462/ceph-osd.1.asok 2020-03-11T22:49:51.793 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:263: TEST_backfill_priority: CEPH_ARGS= 2020-03-11T22:49:51.793 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:263: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:49:51.894 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2020-03-11T22:49:51.895 INFO:tasks.workunit.client.0.smithi059.stdout: "local_reservations": { 2020-03-11T22:49:51.895 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:49:51.895 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:49:51.895 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [ 2020-03-11T22:49:51.895 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:49:51.895 INFO:tasks.workunit.client.0.smithi059.stdout: "priority": 110, 2020-03-11T22:49:51.895 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2020-03-11T22:49:51.896 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:49:51.896 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "1.0", 2020-03-11T22:49:51.896 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 110, 2020-03-11T22:49:51.896 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:49:51.896 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:49:51.896 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:49:51.896 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:49:51.897 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:49:51.897 INFO:tasks.workunit.client.0.smithi059.stdout: "priority": 151, 2020-03-11T22:49:51.897 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2020-03-11T22:49:51.897 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:49:51.897 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "7.0", 2020-03-11T22:49:51.897 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 151, 2020-03-11T22:49:51.897 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:49:51.897 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:49:51.898 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:49:51.898 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:49:51.898 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:49:51.898 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [ 2020-03-11T22:49:51.898 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:49:51.898 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "3.0", 2020-03-11T22:49:51.898 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 151, 2020-03-11T22:49:51.899 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:49:51.899 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:49:51.899 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:49:51.899 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:49:51.899 INFO:tasks.workunit.client.0.smithi059.stdout: "remote_reservations": { 2020-03-11T22:49:51.899 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:49:51.899 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:49:51.900 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [], 2020-03-11T22:49:51.900 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [] 2020-03-11T22:49:51.900 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:49:51.900 INFO:tasks.workunit.client.0.smithi059.stdout:} 2020-03-11T22:49:51.902 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:266: TEST_backfill_priority: flush_pg_stats 2020-03-11T22:49:51.903 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:49:51.903 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:49:52.271 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:49:52.272 INFO:tasks.workunit.client.0.smithi059.stderr:1 2020-03-11T22:49:52.272 INFO:tasks.workunit.client.0.smithi059.stderr:2 2020-03-11T22:49:52.272 INFO:tasks.workunit.client.0.smithi059.stderr:3 2020-03-11T22:49:52.272 INFO:tasks.workunit.client.0.smithi059.stderr:4' 2020-03-11T22:49:52.272 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:49:52.272 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:49:52.272 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:49:52.437 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836528 2020-03-11T22:49:52.437 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836528' 2020-03-11T22:49:52.437 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:49:52.437 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:49:52.569 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949673007 2020-03-11T22:49:52.569 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836528 1-42949673007' 2020-03-11T22:49:52.570 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:49:52.570 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:49:52.702 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509487 2020-03-11T22:49:52.702 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836528 1-42949673007 2-64424509487' 2020-03-11T22:49:52.702 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:49:52.703 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:49:52.834 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=85899345964 2020-03-11T22:49:52.834 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836528 1-42949673007 2-64424509487 3-85899345964' 2020-03-11T22:49:52.834 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:49:52.834 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:49:52.967 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=107374182443 2020-03-11T22:49:52.967 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836528 1-42949673007 2-64424509487 3-85899345964 4-107374182443' 2020-03-11T22:49:52.967 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:49:52.967 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836528 2020-03-11T22:49:52.968 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:49:52.968 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:49:52.969 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836528 2020-03-11T22:49:52.969 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:49:52.970 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836528 2020-03-11T22:49:52.970 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836528' 2020-03-11T22:49:52.971 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:49:52.971 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.0 seq 21474836528 2020-03-11T22:49:53.334 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836526 -lt 21474836528 2020-03-11T22:49:53.335 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:49:54.336 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:49:54.336 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:49:54.712 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836528 -lt 21474836528 2020-03-11T22:49:54.961 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:49:54.962 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949673007 2020-03-11T22:49:54.962 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:49:54.962 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:49:54.963 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949673007 2020-03-11T22:49:54.963 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:49:54.963 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949673007 2020-03-11T22:49:54.963 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949673007' 2020-03-11T22:49:54.963 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:49:54.963 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.1 seq 42949673007 2020-03-11T22:49:55.083 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949673007 -lt 42949673007 2020-03-11T22:49:55.083 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:49:55.084 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509487 2020-03-11T22:49:55.084 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:49:55.084 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:49:55.085 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509487 2020-03-11T22:49:55.085 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:49:55.086 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509487 2020-03-11T22:49:55.086 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509487' 2020-03-11T22:49:55.086 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.2 seq 64424509487 2020-03-11T22:49:55.087 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:49:55.468 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509487 -lt 64424509487 2020-03-11T22:49:55.468 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:49:55.469 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-85899345964 2020-03-11T22:49:55.469 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:49:55.469 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:49:55.470 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-85899345964 2020-03-11T22:49:55.470 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:49:55.471 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=85899345964 2020-03-11T22:49:55.471 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 85899345964' 2020-03-11T22:49:55.471 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:49:55.471 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.3 seq 85899345964 2020-03-11T22:49:55.858 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 85899345965 -lt 85899345964 2020-03-11T22:49:55.858 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:49:55.858 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-107374182443 2020-03-11T22:49:55.858 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:49:55.859 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:49:55.860 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-107374182443 2020-03-11T22:49:55.860 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:49:55.861 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=107374182443 2020-03-11T22:49:55.861 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 107374182443' 2020-03-11T22:49:55.861 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:49:55.862 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.4 seq 107374182443 2020-03-11T22:49:56.228 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 107374182443 -lt 107374182443 2020-03-11T22:49:56.228 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:267: TEST_backfill_priority: ceph pg force-backfill 7.0 2020-03-11T22:49:56.579 INFO:tasks.workunit.client.0.smithi059.stderr:instructing pg(s) [7.0] on osd.1 to force-backfill; 2020-03-11T22:49:56.580 INFO:tasks.workunit.client.0.smithi059.stderr: 2020-03-11T22:49:56.587 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:268: TEST_backfill_priority: sleep 2 2020-03-11T22:49:58.590 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:270: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:49:58.590 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:49:58.590 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:49:58.590 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:49:58.590 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:49:58.591 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56462 2020-03-11T22:49:58.591 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56462/ceph-osd.1.asok 2020-03-11T22:49:58.594 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:270: TEST_backfill_priority: CEPH_ARGS= 2020-03-11T22:49:58.594 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:270: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:49:58.701 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:271: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:49:58.702 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2020-03-11T22:49:58.702 INFO:tasks.workunit.client.0.smithi059.stdout: "local_reservations": { 2020-03-11T22:49:58.702 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:49:58.703 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:49:58.703 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [ 2020-03-11T22:49:58.703 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:49:58.703 INFO:tasks.workunit.client.0.smithi059.stdout: "priority": 110, 2020-03-11T22:49:58.703 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2020-03-11T22:49:58.703 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:49:58.703 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "1.0", 2020-03-11T22:49:58.704 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 110, 2020-03-11T22:49:58.704 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:49:58.704 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:49:58.704 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:49:58.704 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:49:58.704 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:49:58.704 INFO:tasks.workunit.client.0.smithi059.stdout: "priority": 151, 2020-03-11T22:49:58.705 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2020-03-11T22:49:58.705 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:49:58.705 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "3.0", 2020-03-11T22:49:58.705 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 151, 2020-03-11T22:49:58.705 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:49:58.705 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:49:58.705 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:49:58.706 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:49:58.706 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:49:58.706 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [ 2020-03-11T22:49:58.706 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:49:58.706 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "7.0", 2020-03-11T22:49:58.707 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 254, 2020-03-11T22:49:58.707 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:49:58.707 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:49:58.707 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:49:58.707 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:49:58.708 INFO:tasks.workunit.client.0.smithi059.stdout: "remote_reservations": { 2020-03-11T22:49:58.708 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:49:58.708 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:49:58.708 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [], 2020-03-11T22:49:58.708 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [] 2020-03-11T22:49:58.709 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:49:58.709 INFO:tasks.workunit.client.0.smithi059.stdout:} 2020-03-11T22:49:58.712 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:272: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:49:58.713 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:272: TEST_backfill_priority: jq '(.local_reservations.queues[].items[] | select(.item == "3.0")).prio' 2020-03-11T22:49:58.713 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:272: TEST_backfill_priority: PRIO=151 2020-03-11T22:49:58.713 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:273: TEST_backfill_priority: '[' 151 '!=' 151 ']' 2020-03-11T22:49:58.713 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:279: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:49:58.713 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:279: TEST_backfill_priority: jq '.local_reservations.in_progress[0].item' 2020-03-11T22:49:58.713 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:279: TEST_backfill_priority: eval 'ITEM="7.0"' 2020-03-11T22:49:58.714 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:279: TEST_backfill_priority: ITEM=7.0 2020-03-11T22:49:58.714 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:280: TEST_backfill_priority: '[' 7.0 '!=' 7.0 ']' 2020-03-11T22:49:58.714 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:285: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:49:58.714 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:285: TEST_backfill_priority: jq '.local_reservations.in_progress[0].prio' 2020-03-11T22:49:58.714 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:285: TEST_backfill_priority: PRIO=254 2020-03-11T22:49:58.714 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:286: TEST_backfill_priority: '[' 254 '!=' 254 ']' 2020-03-11T22:49:58.714 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:293: TEST_backfill_priority: ceph osd unset noout 2020-03-11T22:49:59.361 INFO:tasks.workunit.client.0.smithi059.stderr:noout is unset 2020-03-11T22:49:59.368 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:294: TEST_backfill_priority: ceph osd unset nobackfill 2020-03-11T22:50:00.134 INFO:tasks.workunit.client.0.smithi059.stderr:nobackfill is unset 2020-03-11T22:50:00.144 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:296: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:50:00.144 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:50:00.144 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:50:00.145 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:50:00.145 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:50:00.145 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56462 2020-03-11T22:50:00.145 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56462/ceph-osd.1.asok 2020-03-11T22:50:00.145 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:296: TEST_backfill_priority: wait_for_clean 'CEPH_ARGS='\'''\'' ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations' 2020-03-11T22:50:00.145 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local 'cmd=CEPH_ARGS='\'''\'' ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations' 2020-03-11T22:50:00.150 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:50:00.150 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:50:00.150 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:50:00.151 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:50:00.151 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:50:00.151 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:50:00.151 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:50:00.151 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:50:00.151 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:50:00.233 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:50:00.233 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:50:00.234 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:50:00.234 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:50:00.234 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:50:00.644 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:50:00.645 INFO:tasks.workunit.client.0.smithi059.stderr:1 2020-03-11T22:50:00.645 INFO:tasks.workunit.client.0.smithi059.stderr:2 2020-03-11T22:50:00.645 INFO:tasks.workunit.client.0.smithi059.stderr:3 2020-03-11T22:50:00.645 INFO:tasks.workunit.client.0.smithi059.stderr:4' 2020-03-11T22:50:00.646 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:50:00.646 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:00.646 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:50:00.780 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836532 2020-03-11T22:50:00.780 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836532' 2020-03-11T22:50:00.780 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:00.780 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:50:00.913 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949673011 2020-03-11T22:50:00.914 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836532 1-42949673011' 2020-03-11T22:50:00.916 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:00.916 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:50:01.046 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509490 2020-03-11T22:50:01.046 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836532 1-42949673011 2-64424509490' 2020-03-11T22:50:01.046 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:01.046 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:50:01.179 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=85899345968 2020-03-11T22:50:01.180 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836532 1-42949673011 2-64424509490 3-85899345968' 2020-03-11T22:50:01.180 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:01.180 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:50:01.314 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=107374182447 2020-03-11T22:50:01.314 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836532 1-42949673011 2-64424509490 3-85899345968 4-107374182447' 2020-03-11T22:50:01.314 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:01.314 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836532 2020-03-11T22:50:01.315 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:01.315 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:50:01.315 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836532 2020-03-11T22:50:01.316 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:01.316 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836532 2020-03-11T22:50:01.316 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836532' 2020-03-11T22:50:01.317 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:50:01.317 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.0 seq 21474836532 2020-03-11T22:50:01.714 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836532 -lt 21474836532 2020-03-11T22:50:01.715 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:01.715 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949673011 2020-03-11T22:50:01.715 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:01.716 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:50:01.716 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949673011 2020-03-11T22:50:01.716 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:01.717 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949673011 2020-03-11T22:50:01.717 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949673011' 2020-03-11T22:50:01.718 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:50:01.718 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.1 seq 42949673011 2020-03-11T22:50:02.166 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949673011 -lt 42949673011 2020-03-11T22:50:02.166 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:02.166 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509490 2020-03-11T22:50:02.167 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:02.167 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:50:02.168 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509490 2020-03-11T22:50:02.168 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:02.168 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509490 2020-03-11T22:50:02.169 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509490' 2020-03-11T22:50:02.169 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.2 seq 64424509490 2020-03-11T22:50:02.169 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:50:02.591 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509490 -lt 64424509490 2020-03-11T22:50:02.591 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:02.592 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-85899345968 2020-03-11T22:50:02.592 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:02.593 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:50:02.593 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-85899345968 2020-03-11T22:50:02.593 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:02.594 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=85899345968 2020-03-11T22:50:02.594 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 85899345968' 2020-03-11T22:50:02.594 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:50:02.594 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.3 seq 85899345968 2020-03-11T22:50:03.016 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 85899345968 -lt 85899345968 2020-03-11T22:50:03.017 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:03.017 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-107374182447 2020-03-11T22:50:03.017 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:03.018 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:50:03.018 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-107374182447 2020-03-11T22:50:03.019 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:03.020 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=107374182447 2020-03-11T22:50:03.020 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 107374182447' 2020-03-11T22:50:03.020 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:50:03.020 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.4 seq 107374182447 2020-03-11T22:50:03.392 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 107374182445 -lt 107374182447 2020-03-11T22:50:03.392 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:50:04.393 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:50:04.394 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:50:04.820 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 107374182447 -lt 107374182447 2020-03-11T22:50:04.821 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:50:04.821 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:50:04.821 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:50:05.331 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 3 == 0 2020-03-11T22:50:05.331 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:50:05.331 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:50:05.331 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:50:05.331 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:50:05.332 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:50:05.332 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:50:05.332 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:50:05.689 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:50:05.689 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:50:05.689 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:50:05.690 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:50:06.165 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:50:06.165 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:50:06.165 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:50:06.165 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:50:06.165 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:06.166 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:50:06.166 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:06.258 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2020-03-11T22:50:06.258 INFO:tasks.workunit.client.0.smithi059.stdout: "local_reservations": { 2020-03-11T22:50:06.258 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:50:06.258 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:50:06.258 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [ 2020-03-11T22:50:06.259 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:06.259 INFO:tasks.workunit.client.0.smithi059.stdout: "priority": 110, 2020-03-11T22:50:06.259 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2020-03-11T22:50:06.259 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:06.259 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "1.0", 2020-03-11T22:50:06.260 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 110, 2020-03-11T22:50:06.260 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:50:06.260 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:06.260 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:50:06.260 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:50:06.260 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:06.260 INFO:tasks.workunit.client.0.smithi059.stdout: "priority": 151, 2020-03-11T22:50:06.261 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2020-03-11T22:50:06.261 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:06.261 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "3.0", 2020-03-11T22:50:06.261 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 151, 2020-03-11T22:50:06.261 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:50:06.261 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:06.262 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:50:06.262 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:06.262 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:50:06.262 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [ 2020-03-11T22:50:06.262 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:06.262 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "7.0", 2020-03-11T22:50:06.263 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 254, 2020-03-11T22:50:06.263 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:50:06.263 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:06.263 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:50:06.263 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:50:06.263 INFO:tasks.workunit.client.0.smithi059.stdout: "remote_reservations": { 2020-03-11T22:50:06.263 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:50:06.264 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:50:06.264 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [], 2020-03-11T22:50:06.264 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [] 2020-03-11T22:50:06.264 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:06.264 INFO:tasks.workunit.client.0.smithi059.stdout:} 2020-03-11T22:50:06.265 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:50:06.366 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:50:06.366 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:50:06.367 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:50:06.367 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:50:06.367 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:50:06.368 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:50:06.368 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:50:06.368 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:50:06.764 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:50:06.765 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:50:06.765 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:50:06.765 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:50:07.245 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:50:07.245 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:50:07.245 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:50:07.245 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:50:07.245 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:50:07.246 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:50:07.246 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:50:07.246 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:50:07.246 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:50:07.788 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=15999072 2020-03-11T22:50:07.788 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 15999072 '!=' null 2020-03-11T22:50:07.789 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:50:07.789 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:07.789 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:50:07.789 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:07.881 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2020-03-11T22:50:07.882 INFO:tasks.workunit.client.0.smithi059.stdout: "local_reservations": { 2020-03-11T22:50:07.882 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:50:07.882 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:50:07.882 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [ 2020-03-11T22:50:07.882 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:07.882 INFO:tasks.workunit.client.0.smithi059.stdout: "priority": 110, 2020-03-11T22:50:07.883 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2020-03-11T22:50:07.883 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:07.883 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "1.0", 2020-03-11T22:50:07.883 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 110, 2020-03-11T22:50:07.883 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:50:07.883 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:07.883 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:50:07.883 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:50:07.884 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:07.884 INFO:tasks.workunit.client.0.smithi059.stdout: "priority": 151, 2020-03-11T22:50:07.884 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2020-03-11T22:50:07.884 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:07.884 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "3.0", 2020-03-11T22:50:07.884 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 151, 2020-03-11T22:50:07.884 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:50:07.885 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:07.885 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:50:07.885 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:07.885 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:50:07.885 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [ 2020-03-11T22:50:07.886 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:07.886 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "7.0", 2020-03-11T22:50:07.886 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 254, 2020-03-11T22:50:07.886 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:50:07.886 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:07.886 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:50:07.887 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:50:07.887 INFO:tasks.workunit.client.0.smithi059.stdout: "remote_reservations": { 2020-03-11T22:50:07.887 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:50:07.888 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:50:07.888 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [], 2020-03-11T22:50:07.888 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [] 2020-03-11T22:50:07.888 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:07.888 INFO:tasks.workunit.client.0.smithi059.stdout:} 2020-03-11T22:50:07.889 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:50:07.992 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:50:07.992 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:50:07.992 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:50:07.992 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:50:07.993 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:50:07.993 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:50:07.993 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:50:07.993 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:50:08.351 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:50:08.352 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:50:08.352 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:50:08.352 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:50:08.845 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:50:08.845 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:50:08.845 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:50:08.845 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:50:08.845 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:50:08.845 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:50:08.846 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:50:08.846 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:50:08.846 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:50:09.320 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=15999072 2020-03-11T22:50:09.320 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 15999072 '!=' null 2020-03-11T22:50:09.320 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:50:09.320 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:09.320 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:50:09.320 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:09.413 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2020-03-11T22:50:09.413 INFO:tasks.workunit.client.0.smithi059.stdout: "local_reservations": { 2020-03-11T22:50:09.413 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:50:09.413 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:50:09.414 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [ 2020-03-11T22:50:09.414 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:09.414 INFO:tasks.workunit.client.0.smithi059.stdout: "priority": 110, 2020-03-11T22:50:09.414 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2020-03-11T22:50:09.414 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:09.414 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "1.0", 2020-03-11T22:50:09.414 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 110, 2020-03-11T22:50:09.415 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:50:09.415 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:09.415 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:50:09.415 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:50:09.415 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:09.415 INFO:tasks.workunit.client.0.smithi059.stdout: "priority": 151, 2020-03-11T22:50:09.415 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2020-03-11T22:50:09.416 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:09.416 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "3.0", 2020-03-11T22:50:09.416 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 151, 2020-03-11T22:50:09.416 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:50:09.416 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:09.416 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:50:09.416 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:09.417 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:50:09.417 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [ 2020-03-11T22:50:09.417 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:09.417 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "7.0", 2020-03-11T22:50:09.417 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 254, 2020-03-11T22:50:09.418 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:50:09.418 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:09.418 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:50:09.418 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:50:09.418 INFO:tasks.workunit.client.0.smithi059.stdout: "remote_reservations": { 2020-03-11T22:50:09.418 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:50:09.419 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:50:09.419 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [], 2020-03-11T22:50:09.419 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [] 2020-03-11T22:50:09.419 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:09.419 INFO:tasks.workunit.client.0.smithi059.stdout:} 2020-03-11T22:50:09.420 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:50:09.522 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:50:09.522 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:50:09.522 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:50:09.522 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:50:09.523 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:50:09.523 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:50:09.523 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:50:09.523 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:50:09.879 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:50:09.880 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:50:09.880 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:50:09.880 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:50:10.359 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:50:10.359 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:50:10.359 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:50:10.359 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:50:10.359 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:50:10.359 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:50:10.360 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:50:10.360 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:50:10.360 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:50:10.920 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=13413482 2020-03-11T22:50:10.920 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 13413482 '!=' null 2020-03-11T22:50:10.920 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:50:10.920 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:10.920 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:50:10.920 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:11.013 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2020-03-11T22:50:11.013 INFO:tasks.workunit.client.0.smithi059.stdout: "local_reservations": { 2020-03-11T22:50:11.013 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:50:11.013 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:50:11.013 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [ 2020-03-11T22:50:11.014 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:11.014 INFO:tasks.workunit.client.0.smithi059.stdout: "priority": 110, 2020-03-11T22:50:11.014 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2020-03-11T22:50:11.014 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:11.014 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "1.0", 2020-03-11T22:50:11.014 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 110, 2020-03-11T22:50:11.014 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:50:11.015 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:11.015 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:50:11.015 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:50:11.015 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:11.015 INFO:tasks.workunit.client.0.smithi059.stdout: "priority": 151, 2020-03-11T22:50:11.015 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2020-03-11T22:50:11.015 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:11.015 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "3.0", 2020-03-11T22:50:11.016 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 151, 2020-03-11T22:50:11.016 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:50:11.016 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:11.016 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:50:11.016 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:11.016 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:50:11.016 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [ 2020-03-11T22:50:11.017 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:11.017 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "7.0", 2020-03-11T22:50:11.017 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 254, 2020-03-11T22:50:11.017 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:50:11.017 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:11.017 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:50:11.018 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:50:11.018 INFO:tasks.workunit.client.0.smithi059.stdout: "remote_reservations": { 2020-03-11T22:50:11.018 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:50:11.018 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:50:11.018 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [], 2020-03-11T22:50:11.018 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [] 2020-03-11T22:50:11.018 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:11.019 INFO:tasks.workunit.client.0.smithi059.stdout:} 2020-03-11T22:50:11.020 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:50:11.122 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:50:11.122 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:50:11.122 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:50:11.122 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:50:11.122 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:50:11.122 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:50:11.123 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:50:11.123 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:50:11.489 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:50:11.489 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:50:11.489 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:50:11.489 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:50:11.965 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:50:11.966 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:50:11.966 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:50:11.966 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:50:11.966 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:50:11.966 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:50:11.966 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:50:11.967 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:50:11.967 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:50:12.449 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=21732369 2020-03-11T22:50:12.449 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 21732369 '!=' null 2020-03-11T22:50:12.449 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:50:12.449 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:12.449 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:50:12.450 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:12.541 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2020-03-11T22:50:12.541 INFO:tasks.workunit.client.0.smithi059.stdout: "local_reservations": { 2020-03-11T22:50:12.541 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:50:12.541 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:50:12.541 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [ 2020-03-11T22:50:12.542 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:12.542 INFO:tasks.workunit.client.0.smithi059.stdout: "priority": 110, 2020-03-11T22:50:12.542 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2020-03-11T22:50:12.542 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:12.542 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "1.0", 2020-03-11T22:50:12.542 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 110, 2020-03-11T22:50:12.542 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:50:12.543 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:12.543 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:50:12.543 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:50:12.543 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:12.543 INFO:tasks.workunit.client.0.smithi059.stdout: "priority": 151, 2020-03-11T22:50:12.543 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2020-03-11T22:50:12.543 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:12.544 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "3.0", 2020-03-11T22:50:12.544 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 151, 2020-03-11T22:50:12.544 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:50:12.544 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:12.544 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:50:12.544 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:12.544 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:50:12.544 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [ 2020-03-11T22:50:12.545 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:12.545 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "7.0", 2020-03-11T22:50:12.545 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 254, 2020-03-11T22:50:12.545 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:50:12.545 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:12.545 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:50:12.545 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:50:12.546 INFO:tasks.workunit.client.0.smithi059.stdout: "remote_reservations": { 2020-03-11T22:50:12.546 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:50:12.546 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:50:12.546 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [], 2020-03-11T22:50:12.546 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [] 2020-03-11T22:50:12.546 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:12.547 INFO:tasks.workunit.client.0.smithi059.stdout:} 2020-03-11T22:50:12.548 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:50:12.649 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:50:12.650 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:50:12.651 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:50:12.651 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:50:12.651 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:50:12.652 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:50:12.652 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:50:12.652 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:50:13.004 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:50:13.004 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:50:13.005 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:50:13.005 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:50:13.485 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:50:13.485 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:50:13.485 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:50:13.486 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:50:13.486 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:50:13.486 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:50:13.486 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:50:13.486 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:50:13.486 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:50:13.963 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=20094410 2020-03-11T22:50:13.963 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 20094410 '!=' null 2020-03-11T22:50:13.963 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:50:13.963 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:13.963 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:50:13.964 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:14.056 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2020-03-11T22:50:14.056 INFO:tasks.workunit.client.0.smithi059.stdout: "local_reservations": { 2020-03-11T22:50:14.056 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:50:14.056 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:50:14.057 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [ 2020-03-11T22:50:14.057 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:14.057 INFO:tasks.workunit.client.0.smithi059.stdout: "priority": 110, 2020-03-11T22:50:14.057 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2020-03-11T22:50:14.057 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:14.057 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "1.0", 2020-03-11T22:50:14.057 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 110, 2020-03-11T22:50:14.058 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:50:14.058 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:14.058 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:50:14.058 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:50:14.058 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:14.058 INFO:tasks.workunit.client.0.smithi059.stdout: "priority": 151, 2020-03-11T22:50:14.058 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2020-03-11T22:50:14.059 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:14.059 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "3.0", 2020-03-11T22:50:14.059 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 151, 2020-03-11T22:50:14.059 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:50:14.059 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:14.059 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:50:14.059 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:14.060 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:50:14.060 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [ 2020-03-11T22:50:14.060 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:14.060 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "7.0", 2020-03-11T22:50:14.060 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 254, 2020-03-11T22:50:14.060 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:50:14.060 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:14.060 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:50:14.061 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:50:14.061 INFO:tasks.workunit.client.0.smithi059.stdout: "remote_reservations": { 2020-03-11T22:50:14.061 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:50:14.061 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:50:14.061 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [], 2020-03-11T22:50:14.062 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [] 2020-03-11T22:50:14.062 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:14.062 INFO:tasks.workunit.client.0.smithi059.stdout:} 2020-03-11T22:50:14.063 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:50:14.165 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:50:14.166 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:50:14.166 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:50:14.166 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:50:14.166 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:50:14.167 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:50:14.167 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:50:14.167 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:50:14.523 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:50:14.524 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:50:14.524 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:50:14.524 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:50:15.003 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:50:15.004 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:50:15.004 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:50:15.004 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:50:15.004 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:50:15.004 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:50:15.004 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:50:15.004 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:50:15.005 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:50:15.565 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=20094403 2020-03-11T22:50:15.566 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 20094403 '!=' null 2020-03-11T22:50:15.566 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:50:15.566 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:15.566 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:50:15.566 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:15.659 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2020-03-11T22:50:15.659 INFO:tasks.workunit.client.0.smithi059.stdout: "local_reservations": { 2020-03-11T22:50:15.659 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:50:15.659 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:50:15.660 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [ 2020-03-11T22:50:15.660 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:15.660 INFO:tasks.workunit.client.0.smithi059.stdout: "priority": 110, 2020-03-11T22:50:15.660 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2020-03-11T22:50:15.660 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:15.660 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "1.0", 2020-03-11T22:50:15.661 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 110, 2020-03-11T22:50:15.661 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:50:15.661 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:15.661 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:50:15.661 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:50:15.661 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:15.661 INFO:tasks.workunit.client.0.smithi059.stdout: "priority": 151, 2020-03-11T22:50:15.662 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2020-03-11T22:50:15.662 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:15.662 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "3.0", 2020-03-11T22:50:15.662 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 151, 2020-03-11T22:50:15.662 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:50:15.662 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:15.662 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:50:15.663 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:15.663 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:50:15.663 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [ 2020-03-11T22:50:15.663 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:15.663 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "7.0", 2020-03-11T22:50:15.663 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 254, 2020-03-11T22:50:15.663 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:50:15.664 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:15.664 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:50:15.664 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:50:15.664 INFO:tasks.workunit.client.0.smithi059.stdout: "remote_reservations": { 2020-03-11T22:50:15.664 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:50:15.664 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:50:15.664 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [], 2020-03-11T22:50:15.665 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [] 2020-03-11T22:50:15.665 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:15.665 INFO:tasks.workunit.client.0.smithi059.stdout:} 2020-03-11T22:50:15.666 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:50:15.768 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:50:15.768 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:50:15.768 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:50:15.768 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:50:15.768 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:50:15.769 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:50:15.769 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:50:15.769 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:50:16.134 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:50:16.134 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:50:16.135 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:50:16.135 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:50:16.623 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:50:16.623 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:50:16.623 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:50:16.623 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:50:16.623 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:50:16.624 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:50:16.624 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:50:16.624 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:50:16.624 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:50:17.155 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=20094403 2020-03-11T22:50:17.156 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 20094403 '!=' null 2020-03-11T22:50:17.156 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:50:17.156 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:17.156 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:50:17.156 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:17.247 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2020-03-11T22:50:17.248 INFO:tasks.workunit.client.0.smithi059.stdout: "local_reservations": { 2020-03-11T22:50:17.248 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:50:17.248 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:50:17.248 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [ 2020-03-11T22:50:17.248 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:17.248 INFO:tasks.workunit.client.0.smithi059.stdout: "priority": 110, 2020-03-11T22:50:17.248 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2020-03-11T22:50:17.249 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:17.249 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "1.0", 2020-03-11T22:50:17.249 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 110, 2020-03-11T22:50:17.249 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:50:17.249 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:17.249 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:50:17.249 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:50:17.249 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:17.250 INFO:tasks.workunit.client.0.smithi059.stdout: "priority": 151, 2020-03-11T22:50:17.250 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2020-03-11T22:50:17.250 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:17.250 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "3.0", 2020-03-11T22:50:17.250 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 151, 2020-03-11T22:50:17.250 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:50:17.250 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:17.251 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:50:17.251 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:17.251 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:50:17.251 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [ 2020-03-11T22:50:17.251 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:17.251 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "7.0", 2020-03-11T22:50:17.251 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 254, 2020-03-11T22:50:17.252 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:50:17.252 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:17.252 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:50:17.252 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:50:17.252 INFO:tasks.workunit.client.0.smithi059.stdout: "remote_reservations": { 2020-03-11T22:50:17.252 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:50:17.252 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:50:17.252 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [], 2020-03-11T22:50:17.253 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [] 2020-03-11T22:50:17.253 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:17.253 INFO:tasks.workunit.client.0.smithi059.stdout:} 2020-03-11T22:50:17.255 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:50:17.356 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:50:17.356 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:50:17.357 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:50:17.357 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:50:17.357 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:50:17.358 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:50:17.358 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:50:17.358 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:50:17.715 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:50:17.716 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:50:17.716 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:50:17.716 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:50:18.202 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:50:18.202 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:50:18.202 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:50:18.202 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:50:18.202 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:50:18.203 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:50:18.203 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:50:18.203 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:50:18.203 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:50:18.687 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=30578423 2020-03-11T22:50:18.687 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 30578423 '!=' null 2020-03-11T22:50:18.687 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:50:18.687 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:18.688 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:50:18.688 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:18.779 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2020-03-11T22:50:18.779 INFO:tasks.workunit.client.0.smithi059.stdout: "local_reservations": { 2020-03-11T22:50:18.779 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:50:18.779 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:50:18.779 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [ 2020-03-11T22:50:18.780 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:18.780 INFO:tasks.workunit.client.0.smithi059.stdout: "priority": 110, 2020-03-11T22:50:18.780 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2020-03-11T22:50:18.780 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:18.780 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "1.0", 2020-03-11T22:50:18.780 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 110, 2020-03-11T22:50:18.780 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:50:18.781 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:18.781 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:50:18.781 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:50:18.781 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:18.781 INFO:tasks.workunit.client.0.smithi059.stdout: "priority": 151, 2020-03-11T22:50:18.781 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2020-03-11T22:50:18.781 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:18.781 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "3.0", 2020-03-11T22:50:18.782 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 151, 2020-03-11T22:50:18.782 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:50:18.782 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:18.782 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:50:18.782 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:18.782 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:50:18.782 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [ 2020-03-11T22:50:18.782 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:18.783 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "7.0", 2020-03-11T22:50:18.783 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 254, 2020-03-11T22:50:18.783 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:50:18.783 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:18.783 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:50:18.783 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:50:18.783 INFO:tasks.workunit.client.0.smithi059.stdout: "remote_reservations": { 2020-03-11T22:50:18.783 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:50:18.784 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:50:18.784 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [], 2020-03-11T22:50:18.784 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [] 2020-03-11T22:50:18.784 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:18.784 INFO:tasks.workunit.client.0.smithi059.stdout:} 2020-03-11T22:50:18.786 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:50:18.887 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:50:18.887 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:50:18.888 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:50:18.889 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:50:18.889 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:50:18.889 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:50:18.889 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:50:18.889 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:50:19.247 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:50:19.247 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:50:19.248 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:50:19.248 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:50:19.764 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:50:19.764 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:50:19.764 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:50:19.764 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:50:19.764 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:50:19.765 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:50:19.765 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:50:19.765 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:50:19.765 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:50:20.260 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=20094331 2020-03-11T22:50:20.260 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 20094331 '!=' null 2020-03-11T22:50:20.260 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:50:20.260 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:20.260 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:50:20.261 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:20.352 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2020-03-11T22:50:20.352 INFO:tasks.workunit.client.0.smithi059.stdout: "local_reservations": { 2020-03-11T22:50:20.352 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:50:20.353 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:50:20.353 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [ 2020-03-11T22:50:20.353 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:20.353 INFO:tasks.workunit.client.0.smithi059.stdout: "priority": 110, 2020-03-11T22:50:20.353 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2020-03-11T22:50:20.353 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:20.353 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "1.0", 2020-03-11T22:50:20.354 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 110, 2020-03-11T22:50:20.354 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:50:20.354 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:20.354 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:50:20.354 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:50:20.354 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:20.354 INFO:tasks.workunit.client.0.smithi059.stdout: "priority": 151, 2020-03-11T22:50:20.354 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2020-03-11T22:50:20.355 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:20.355 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "3.0", 2020-03-11T22:50:20.355 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 151, 2020-03-11T22:50:20.355 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:50:20.355 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:20.355 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:50:20.355 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:20.355 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:50:20.356 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [ 2020-03-11T22:50:20.356 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:20.356 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "7.0", 2020-03-11T22:50:20.356 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 254, 2020-03-11T22:50:20.356 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:50:20.356 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:20.356 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:50:20.357 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:50:20.357 INFO:tasks.workunit.client.0.smithi059.stdout: "remote_reservations": { 2020-03-11T22:50:20.357 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:50:20.357 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:50:20.357 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [], 2020-03-11T22:50:20.357 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [] 2020-03-11T22:50:20.357 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:20.358 INFO:tasks.workunit.client.0.smithi059.stdout:} 2020-03-11T22:50:20.359 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:50:20.460 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:50:20.460 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:50:20.461 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:50:20.462 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:50:20.462 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:50:20.462 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:50:20.462 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:50:20.462 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:50:20.822 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:50:20.822 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:50:20.822 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:50:20.823 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:50:21.314 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:50:21.314 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:50:21.314 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:50:21.315 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:50:21.315 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:50:21.315 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:50:21.315 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:50:21.315 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:50:21.315 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:50:21.795 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=30578220 2020-03-11T22:50:21.796 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 30578220 '!=' null 2020-03-11T22:50:21.796 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:50:21.796 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:21.796 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:50:21.796 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:21.888 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2020-03-11T22:50:21.888 INFO:tasks.workunit.client.0.smithi059.stdout: "local_reservations": { 2020-03-11T22:50:21.889 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:50:21.889 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:50:21.889 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [ 2020-03-11T22:50:21.889 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:21.889 INFO:tasks.workunit.client.0.smithi059.stdout: "priority": 110, 2020-03-11T22:50:21.889 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2020-03-11T22:50:21.889 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:21.890 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "1.0", 2020-03-11T22:50:21.890 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 110, 2020-03-11T22:50:21.890 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:50:21.890 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:21.890 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:50:21.890 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:50:21.890 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:21.891 INFO:tasks.workunit.client.0.smithi059.stdout: "priority": 151, 2020-03-11T22:50:21.891 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2020-03-11T22:50:21.891 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:21.891 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "3.0", 2020-03-11T22:50:21.891 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 151, 2020-03-11T22:50:21.891 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:50:21.891 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:21.891 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:50:21.892 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:21.892 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:50:21.892 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [ 2020-03-11T22:50:21.892 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:21.892 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "7.0", 2020-03-11T22:50:21.892 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 254, 2020-03-11T22:50:21.892 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:50:21.892 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:21.893 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:50:21.893 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:50:21.893 INFO:tasks.workunit.client.0.smithi059.stdout: "remote_reservations": { 2020-03-11T22:50:21.893 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:50:21.893 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:50:21.893 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [], 2020-03-11T22:50:21.893 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [] 2020-03-11T22:50:21.894 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:21.894 INFO:tasks.workunit.client.0.smithi059.stdout:} 2020-03-11T22:50:21.895 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:50:21.997 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:50:21.997 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:50:21.998 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:50:21.998 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:50:21.998 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:50:21.998 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:50:21.999 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:50:21.999 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:50:22.361 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:50:22.362 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:50:22.362 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:50:22.362 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:50:22.907 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:50:22.907 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:50:22.907 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:50:22.907 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:50:22.907 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:50:22.907 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:50:22.908 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:50:22.908 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:50:22.908 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:50:23.563 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=31451762 2020-03-11T22:50:23.564 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 31451762 '!=' null 2020-03-11T22:50:23.564 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:50:23.564 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:23.564 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:50:23.565 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:23.655 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2020-03-11T22:50:23.656 INFO:tasks.workunit.client.0.smithi059.stdout: "local_reservations": { 2020-03-11T22:50:23.656 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:50:23.656 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:50:23.656 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [ 2020-03-11T22:50:23.656 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:23.657 INFO:tasks.workunit.client.0.smithi059.stdout: "priority": 110, 2020-03-11T22:50:23.657 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2020-03-11T22:50:23.657 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:23.657 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "1.0", 2020-03-11T22:50:23.657 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 110, 2020-03-11T22:50:23.657 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:50:23.658 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:23.658 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:50:23.658 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:23.658 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:50:23.658 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [ 2020-03-11T22:50:23.658 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:23.658 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "3.0", 2020-03-11T22:50:23.659 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 151, 2020-03-11T22:50:23.660 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:50:23.660 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:23.660 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:50:23.660 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:50:23.660 INFO:tasks.workunit.client.0.smithi059.stdout: "remote_reservations": { 2020-03-11T22:50:23.660 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:50:23.661 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:50:23.661 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [], 2020-03-11T22:50:23.661 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [] 2020-03-11T22:50:23.661 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:23.661 INFO:tasks.workunit.client.0.smithi059.stdout:} 2020-03-11T22:50:23.663 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:50:23.764 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:50:23.764 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:50:23.765 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:50:23.765 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:50:23.765 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:50:23.765 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:50:23.765 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:50:23.766 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:50:24.125 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:50:24.126 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:50:24.126 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:50:24.126 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:50:24.608 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:50:24.608 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:50:24.608 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:50:24.608 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:50:24.608 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:50:24.608 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:50:24.609 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:50:24.609 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:50:24.609 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:50:25.182 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=31451762 2020-03-11T22:50:25.182 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 31451762 '!=' null 2020-03-11T22:50:25.183 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:50:25.183 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:25.183 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:50:25.183 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:25.275 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2020-03-11T22:50:25.275 INFO:tasks.workunit.client.0.smithi059.stdout: "local_reservations": { 2020-03-11T22:50:25.276 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:50:25.276 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:50:25.276 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [ 2020-03-11T22:50:25.276 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:25.276 INFO:tasks.workunit.client.0.smithi059.stdout: "priority": 110, 2020-03-11T22:50:25.276 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2020-03-11T22:50:25.276 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:25.277 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "1.0", 2020-03-11T22:50:25.277 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 110, 2020-03-11T22:50:25.277 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:50:25.277 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:25.277 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:50:25.277 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:25.277 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:50:25.278 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [ 2020-03-11T22:50:25.278 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:25.278 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "3.0", 2020-03-11T22:50:25.278 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 151, 2020-03-11T22:50:25.278 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:50:25.278 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:25.279 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:50:25.279 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:50:25.279 INFO:tasks.workunit.client.0.smithi059.stdout: "remote_reservations": { 2020-03-11T22:50:25.279 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:50:25.279 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:50:25.279 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [], 2020-03-11T22:50:25.279 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [] 2020-03-11T22:50:25.280 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:25.280 INFO:tasks.workunit.client.0.smithi059.stdout:} 2020-03-11T22:50:25.282 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:50:25.384 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:50:25.384 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:50:25.385 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:50:25.385 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:50:25.385 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:50:25.385 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:50:25.385 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:50:25.385 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:50:25.748 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:50:25.748 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:50:25.749 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:50:25.749 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:50:26.226 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:50:26.226 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:50:26.226 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:50:26.226 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:50:26.227 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:50:26.227 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:50:26.227 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:50:26.227 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:50:26.227 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:50:26.768 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=15725900 2020-03-11T22:50:26.768 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 15725900 '!=' null 2020-03-11T22:50:26.769 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:50:26.769 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:26.769 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:50:26.769 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:26.862 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2020-03-11T22:50:26.862 INFO:tasks.workunit.client.0.smithi059.stdout: "local_reservations": { 2020-03-11T22:50:26.862 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:50:26.862 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:50:26.862 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [ 2020-03-11T22:50:26.862 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:26.863 INFO:tasks.workunit.client.0.smithi059.stdout: "priority": 110, 2020-03-11T22:50:26.863 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2020-03-11T22:50:26.863 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:26.863 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "1.0", 2020-03-11T22:50:26.863 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 110, 2020-03-11T22:50:26.863 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:50:26.863 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:26.864 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:50:26.864 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:26.864 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:50:26.864 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [ 2020-03-11T22:50:26.864 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:26.864 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "3.0", 2020-03-11T22:50:26.864 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 151, 2020-03-11T22:50:26.864 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:50:26.865 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:26.865 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:50:26.865 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:50:26.865 INFO:tasks.workunit.client.0.smithi059.stdout: "remote_reservations": { 2020-03-11T22:50:26.865 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:50:26.865 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:50:26.865 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [], 2020-03-11T22:50:26.866 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [] 2020-03-11T22:50:26.866 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:26.866 INFO:tasks.workunit.client.0.smithi059.stdout:} 2020-03-11T22:50:26.869 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:50:26.970 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:50:26.970 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:50:26.971 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:50:26.971 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:50:26.971 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:50:26.971 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:50:26.971 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:50:26.972 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:50:27.330 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:50:27.330 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:50:27.330 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:50:27.331 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:50:27.808 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:50:27.808 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 0 2020-03-11T22:50:27.808 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:50:27.808 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=1 2020-03-11T22:50:27.809 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:27.809 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:50:27.809 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:27.902 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2020-03-11T22:50:27.902 INFO:tasks.workunit.client.0.smithi059.stdout: "local_reservations": { 2020-03-11T22:50:27.903 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:50:27.903 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:50:27.903 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [ 2020-03-11T22:50:27.903 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:27.903 INFO:tasks.workunit.client.0.smithi059.stdout: "priority": 110, 2020-03-11T22:50:27.904 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2020-03-11T22:50:27.904 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:27.904 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "1.0", 2020-03-11T22:50:27.904 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 110, 2020-03-11T22:50:27.904 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:50:27.904 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:27.904 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:50:27.905 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:27.905 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:50:27.905 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [ 2020-03-11T22:50:27.905 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:27.905 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "3.0", 2020-03-11T22:50:27.905 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 151, 2020-03-11T22:50:27.906 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:50:27.906 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:27.906 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:50:27.906 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:50:27.906 INFO:tasks.workunit.client.0.smithi059.stdout: "remote_reservations": { 2020-03-11T22:50:27.906 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:50:27.906 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:50:27.906 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [], 2020-03-11T22:50:27.907 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [] 2020-03-11T22:50:27.907 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:27.907 INFO:tasks.workunit.client.0.smithi059.stdout:} 2020-03-11T22:50:27.909 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:50:28.010 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:50:28.011 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:50:28.012 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:50:28.012 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:50:28.012 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:50:28.012 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:50:28.012 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:50:28.012 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:50:28.367 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:50:28.367 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:50:28.368 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:50:28.368 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:50:28.858 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:50:28.859 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:50:28.859 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:50:28.859 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:50:28.859 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:50:28.859 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:50:28.860 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:50:28.860 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:50:28.860 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:50:29.338 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=30141371 2020-03-11T22:50:29.339 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 30141371 '!=' null 2020-03-11T22:50:29.339 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:50:29.339 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:29.339 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:50:29.340 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:29.430 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2020-03-11T22:50:29.430 INFO:tasks.workunit.client.0.smithi059.stdout: "local_reservations": { 2020-03-11T22:50:29.430 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:50:29.431 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:50:29.431 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [ 2020-03-11T22:50:29.431 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:29.431 INFO:tasks.workunit.client.0.smithi059.stdout: "priority": 110, 2020-03-11T22:50:29.432 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2020-03-11T22:50:29.432 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:29.433 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "1.0", 2020-03-11T22:50:29.433 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 110, 2020-03-11T22:50:29.433 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:50:29.434 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:29.434 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:50:29.434 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:29.434 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:50:29.435 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [ 2020-03-11T22:50:29.435 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:29.435 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "3.0", 2020-03-11T22:50:29.436 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 151, 2020-03-11T22:50:29.436 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:50:29.436 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:29.436 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:50:29.437 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:50:29.437 INFO:tasks.workunit.client.0.smithi059.stdout: "remote_reservations": { 2020-03-11T22:50:29.437 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:50:29.437 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:50:29.438 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [], 2020-03-11T22:50:29.438 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [] 2020-03-11T22:50:29.438 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:29.438 INFO:tasks.workunit.client.0.smithi059.stdout:} 2020-03-11T22:50:29.439 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:50:29.538 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:50:29.538 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:50:29.539 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:50:29.540 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:50:29.540 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:50:29.540 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:50:29.540 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:50:29.540 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:50:29.898 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:50:29.898 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:50:29.898 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:50:29.899 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:50:30.383 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:50:30.383 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:50:30.383 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:50:30.384 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:50:30.384 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:50:30.384 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:50:30.384 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:50:30.384 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:50:30.384 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:50:30.861 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=14415485 2020-03-11T22:50:30.861 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 14415485 '!=' null 2020-03-11T22:50:30.861 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:50:30.861 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:30.862 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:50:30.862 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:30.954 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2020-03-11T22:50:30.954 INFO:tasks.workunit.client.0.smithi059.stdout: "local_reservations": { 2020-03-11T22:50:30.954 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:50:30.955 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:50:30.955 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [ 2020-03-11T22:50:30.955 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:30.955 INFO:tasks.workunit.client.0.smithi059.stdout: "priority": 110, 2020-03-11T22:50:30.955 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2020-03-11T22:50:30.955 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:30.955 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "1.0", 2020-03-11T22:50:30.956 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 110, 2020-03-11T22:50:30.956 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:50:30.956 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:30.956 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:50:30.956 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:30.956 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:50:30.956 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [ 2020-03-11T22:50:30.957 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:30.957 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "3.0", 2020-03-11T22:50:30.957 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 151, 2020-03-11T22:50:30.957 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:50:30.957 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:30.957 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:50:30.957 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:50:30.958 INFO:tasks.workunit.client.0.smithi059.stdout: "remote_reservations": { 2020-03-11T22:50:30.958 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:50:30.958 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:50:30.958 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [], 2020-03-11T22:50:30.958 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [] 2020-03-11T22:50:30.958 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:30.958 INFO:tasks.workunit.client.0.smithi059.stdout:} 2020-03-11T22:50:30.961 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:50:31.063 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:50:31.064 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:50:31.065 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:50:31.065 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:50:31.065 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:50:31.065 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:50:31.065 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:50:31.065 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:50:31.422 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:50:31.423 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:50:31.423 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:50:31.423 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:50:31.900 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:50:31.900 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:50:31.900 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:50:31.900 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:50:31.901 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:50:31.901 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:50:31.901 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:50:31.901 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:50:31.901 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:50:32.425 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=26693708 2020-03-11T22:50:32.425 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 26693708 '!=' null 2020-03-11T22:50:32.426 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:50:32.426 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:32.426 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:50:32.426 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:32.517 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2020-03-11T22:50:32.517 INFO:tasks.workunit.client.0.smithi059.stdout: "local_reservations": { 2020-03-11T22:50:32.517 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:50:32.517 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:50:32.518 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [ 2020-03-11T22:50:32.518 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:32.518 INFO:tasks.workunit.client.0.smithi059.stdout: "priority": 110, 2020-03-11T22:50:32.518 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2020-03-11T22:50:32.518 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:32.518 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "1.0", 2020-03-11T22:50:32.518 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 110, 2020-03-11T22:50:32.519 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:50:32.519 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:32.519 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:50:32.519 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:32.519 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:50:32.519 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [ 2020-03-11T22:50:32.519 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:32.520 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "3.0", 2020-03-11T22:50:32.520 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 151, 2020-03-11T22:50:32.520 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:50:32.520 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:32.520 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:50:32.520 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:50:32.520 INFO:tasks.workunit.client.0.smithi059.stdout: "remote_reservations": { 2020-03-11T22:50:32.520 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:50:32.521 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:50:32.521 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [], 2020-03-11T22:50:32.521 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [] 2020-03-11T22:50:32.521 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:32.521 INFO:tasks.workunit.client.0.smithi059.stdout:} 2020-03-11T22:50:32.524 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:50:32.626 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:50:32.626 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:50:32.627 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:50:32.627 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:50:32.627 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:50:32.627 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:50:32.627 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:50:32.627 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:50:32.985 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:50:32.985 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:50:32.986 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:50:32.986 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:50:33.600 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:50:33.600 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:50:33.601 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:50:33.601 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:50:33.601 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:50:33.601 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:50:33.602 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:50:33.602 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:50:33.602 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:50:34.085 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=22978010 2020-03-11T22:50:34.085 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 22978010 '!=' null 2020-03-11T22:50:34.085 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:50:34.086 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:34.086 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:50:34.086 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:34.178 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2020-03-11T22:50:34.178 INFO:tasks.workunit.client.0.smithi059.stdout: "local_reservations": { 2020-03-11T22:50:34.178 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:50:34.178 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:50:34.178 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [ 2020-03-11T22:50:34.179 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:34.179 INFO:tasks.workunit.client.0.smithi059.stdout: "priority": 110, 2020-03-11T22:50:34.179 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2020-03-11T22:50:34.179 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:34.179 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "1.0", 2020-03-11T22:50:34.179 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 110, 2020-03-11T22:50:34.180 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:50:34.180 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:34.180 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:50:34.180 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:34.180 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:50:34.180 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [ 2020-03-11T22:50:34.180 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:34.181 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "3.0", 2020-03-11T22:50:34.181 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 151, 2020-03-11T22:50:34.181 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:50:34.181 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:34.181 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:50:34.181 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:50:34.181 INFO:tasks.workunit.client.0.smithi059.stdout: "remote_reservations": { 2020-03-11T22:50:34.182 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:50:34.182 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:50:34.182 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [], 2020-03-11T22:50:34.182 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [] 2020-03-11T22:50:34.182 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:34.182 INFO:tasks.workunit.client.0.smithi059.stdout:} 2020-03-11T22:50:34.185 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:50:34.287 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:50:34.287 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:50:34.287 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:50:34.287 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:50:34.288 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:50:34.288 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:50:34.288 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:50:34.288 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:50:34.646 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:50:34.647 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:50:34.647 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:50:34.647 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:50:35.119 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:50:35.120 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:50:35.120 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:50:35.120 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:50:35.120 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:50:35.120 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:50:35.120 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:50:35.121 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:50:35.121 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:50:35.596 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=20094132 2020-03-11T22:50:35.596 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 20094132 '!=' null 2020-03-11T22:50:35.596 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:50:35.596 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:35.596 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:50:35.597 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:35.688 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2020-03-11T22:50:35.688 INFO:tasks.workunit.client.0.smithi059.stdout: "local_reservations": { 2020-03-11T22:50:35.688 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:50:35.688 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:50:35.688 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [ 2020-03-11T22:50:35.689 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:35.689 INFO:tasks.workunit.client.0.smithi059.stdout: "priority": 110, 2020-03-11T22:50:35.689 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2020-03-11T22:50:35.689 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:35.689 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "1.0", 2020-03-11T22:50:35.689 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 110, 2020-03-11T22:50:35.689 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:50:35.690 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:35.690 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:50:35.690 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:35.690 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:50:35.690 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [ 2020-03-11T22:50:35.690 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:35.691 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "3.0", 2020-03-11T22:50:35.691 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 151, 2020-03-11T22:50:35.691 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:50:35.691 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:35.691 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:50:35.691 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:50:35.691 INFO:tasks.workunit.client.0.smithi059.stdout: "remote_reservations": { 2020-03-11T22:50:35.692 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:50:35.692 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:50:35.692 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [], 2020-03-11T22:50:35.692 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [] 2020-03-11T22:50:35.692 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:35.692 INFO:tasks.workunit.client.0.smithi059.stdout:} 2020-03-11T22:50:35.695 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:50:35.797 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:50:35.797 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:50:35.797 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:50:35.798 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:50:35.798 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:50:35.798 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:50:35.798 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:50:35.798 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:50:36.156 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:50:36.157 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:50:36.157 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:50:36.157 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:50:36.648 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:50:36.648 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:50:36.648 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:50:36.649 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:50:36.649 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:50:36.649 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:50:36.649 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:50:36.649 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:50:36.649 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:50:37.143 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=20094132 2020-03-11T22:50:37.143 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 20094132 '!=' null 2020-03-11T22:50:37.144 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:50:37.144 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:37.144 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:50:37.144 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:37.235 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2020-03-11T22:50:37.236 INFO:tasks.workunit.client.0.smithi059.stdout: "local_reservations": { 2020-03-11T22:50:37.236 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:50:37.236 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:50:37.236 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [ 2020-03-11T22:50:37.237 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:37.237 INFO:tasks.workunit.client.0.smithi059.stdout: "priority": 110, 2020-03-11T22:50:37.237 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2020-03-11T22:50:37.237 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:37.237 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "1.0", 2020-03-11T22:50:37.238 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 110, 2020-03-11T22:50:37.238 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:50:37.238 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:37.238 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:50:37.239 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:37.239 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:50:37.239 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [ 2020-03-11T22:50:37.239 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:37.239 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "3.0", 2020-03-11T22:50:37.240 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 151, 2020-03-11T22:50:37.240 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:50:37.240 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:37.240 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:50:37.241 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:50:37.241 INFO:tasks.workunit.client.0.smithi059.stdout: "remote_reservations": { 2020-03-11T22:50:37.241 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:50:37.241 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:50:37.241 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [], 2020-03-11T22:50:37.242 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [] 2020-03-11T22:50:37.242 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:37.242 INFO:tasks.workunit.client.0.smithi059.stdout:} 2020-03-11T22:50:37.243 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:50:37.345 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:50:37.345 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:50:37.345 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:50:37.346 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:50:37.346 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:50:37.346 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:50:37.346 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:50:37.346 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:50:37.712 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:50:37.713 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:50:37.713 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:50:37.713 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:50:38.201 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:50:38.201 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:50:38.202 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:50:38.202 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:50:38.202 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:50:38.202 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:50:38.202 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:50:38.202 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:50:38.202 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:50:38.768 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=30578009 2020-03-11T22:50:38.768 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 30578009 '!=' null 2020-03-11T22:50:38.768 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:50:38.768 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:38.768 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:50:38.769 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:38.860 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2020-03-11T22:50:38.861 INFO:tasks.workunit.client.0.smithi059.stdout: "local_reservations": { 2020-03-11T22:50:38.861 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:50:38.861 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:50:38.861 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [ 2020-03-11T22:50:38.861 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:38.861 INFO:tasks.workunit.client.0.smithi059.stdout: "priority": 110, 2020-03-11T22:50:38.862 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2020-03-11T22:50:38.862 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:38.862 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "1.0", 2020-03-11T22:50:38.862 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 110, 2020-03-11T22:50:38.862 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:50:38.862 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:38.862 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:50:38.863 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:38.863 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:50:38.863 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [ 2020-03-11T22:50:38.863 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:38.863 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "3.0", 2020-03-11T22:50:38.863 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 151, 2020-03-11T22:50:38.863 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:50:38.864 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:38.864 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:50:38.864 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:50:38.864 INFO:tasks.workunit.client.0.smithi059.stdout: "remote_reservations": { 2020-03-11T22:50:38.864 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:50:38.864 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:50:38.864 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [], 2020-03-11T22:50:38.865 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [] 2020-03-11T22:50:38.865 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:38.865 INFO:tasks.workunit.client.0.smithi059.stdout:} 2020-03-11T22:50:38.869 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:50:38.969 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:50:38.970 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:50:38.971 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:50:38.971 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:50:38.971 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:50:38.971 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:50:38.972 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:50:38.972 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:50:39.333 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:50:39.333 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:50:39.334 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:50:39.334 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:50:39.816 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:50:39.816 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:50:39.817 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:50:39.817 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:50:39.817 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:50:39.817 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:50:39.817 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:50:39.817 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:50:39.818 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:50:40.347 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=20967692 2020-03-11T22:50:40.348 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 20967692 '!=' null 2020-03-11T22:50:40.348 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:50:40.348 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:40.348 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:50:40.348 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:40.440 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2020-03-11T22:50:40.440 INFO:tasks.workunit.client.0.smithi059.stdout: "local_reservations": { 2020-03-11T22:50:40.440 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:50:40.440 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:50:40.440 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [ 2020-03-11T22:50:40.441 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:40.441 INFO:tasks.workunit.client.0.smithi059.stdout: "priority": 110, 2020-03-11T22:50:40.441 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2020-03-11T22:50:40.441 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:40.441 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "1.0", 2020-03-11T22:50:40.442 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 110, 2020-03-11T22:50:40.442 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:50:40.442 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:40.442 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:50:40.442 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:40.442 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:50:40.442 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [ 2020-03-11T22:50:40.443 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:40.443 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "3.0", 2020-03-11T22:50:40.443 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 151, 2020-03-11T22:50:40.443 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:50:40.443 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:40.443 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:50:40.443 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:50:40.443 INFO:tasks.workunit.client.0.smithi059.stdout: "remote_reservations": { 2020-03-11T22:50:40.444 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:50:40.444 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:50:40.444 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [], 2020-03-11T22:50:40.444 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [] 2020-03-11T22:50:40.444 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:40.444 INFO:tasks.workunit.client.0.smithi059.stdout:} 2020-03-11T22:50:40.447 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:50:40.548 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:50:40.549 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:50:40.549 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:50:40.549 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:50:40.549 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:50:40.549 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:50:40.549 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:50:40.550 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:50:40.906 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:50:40.907 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:50:40.907 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:50:40.907 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:50:41.436 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:50:41.437 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:50:41.437 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:50:41.437 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:50:41.437 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:50:41.437 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:50:41.437 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:50:41.438 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:50:41.438 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:50:42.001 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=32325095 2020-03-11T22:50:42.002 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 32325095 '!=' null 2020-03-11T22:50:42.002 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:50:42.002 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:42.002 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:50:42.002 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:42.094 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2020-03-11T22:50:42.095 INFO:tasks.workunit.client.0.smithi059.stdout: "local_reservations": { 2020-03-11T22:50:42.095 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:50:42.095 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:50:42.095 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [ 2020-03-11T22:50:42.095 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:42.095 INFO:tasks.workunit.client.0.smithi059.stdout: "priority": 110, 2020-03-11T22:50:42.095 INFO:tasks.workunit.client.0.smithi059.stdout: "items": [ 2020-03-11T22:50:42.095 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:42.096 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "1.0", 2020-03-11T22:50:42.096 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 110, 2020-03-11T22:50:42.096 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:50:42.096 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:42.096 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:50:42.096 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:42.097 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:50:42.097 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [ 2020-03-11T22:50:42.097 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:42.097 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "3.0", 2020-03-11T22:50:42.097 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 151, 2020-03-11T22:50:42.097 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:50:42.097 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:42.098 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:50:42.098 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:50:42.098 INFO:tasks.workunit.client.0.smithi059.stdout: "remote_reservations": { 2020-03-11T22:50:42.098 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:50:42.098 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:50:42.098 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [], 2020-03-11T22:50:42.098 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [] 2020-03-11T22:50:42.099 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:42.099 INFO:tasks.workunit.client.0.smithi059.stdout:} 2020-03-11T22:50:42.101 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:50:42.203 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:50:42.203 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:50:42.203 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:50:42.203 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:50:42.204 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:50:42.204 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:50:42.204 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:50:42.204 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:50:42.564 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:50:42.565 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:50:42.565 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:50:42.565 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:50:43.039 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:50:43.040 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:50:43.040 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:50:43.040 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:50:43.040 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:50:43.040 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:50:43.040 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:50:43.040 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:50:43.041 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:50:43.661 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=32325095 2020-03-11T22:50:43.661 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 32325095 '!=' null 2020-03-11T22:50:43.661 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:50:43.661 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:43.662 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:50:43.662 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:43.754 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2020-03-11T22:50:43.754 INFO:tasks.workunit.client.0.smithi059.stdout: "local_reservations": { 2020-03-11T22:50:43.754 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:50:43.754 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:50:43.754 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [], 2020-03-11T22:50:43.754 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [ 2020-03-11T22:50:43.755 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:43.755 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "1.0", 2020-03-11T22:50:43.755 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 110, 2020-03-11T22:50:43.755 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:50:43.755 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:43.755 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:50:43.755 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:50:43.756 INFO:tasks.workunit.client.0.smithi059.stdout: "remote_reservations": { 2020-03-11T22:50:43.756 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:50:43.756 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:50:43.756 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [], 2020-03-11T22:50:43.756 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [] 2020-03-11T22:50:43.756 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:43.756 INFO:tasks.workunit.client.0.smithi059.stdout:} 2020-03-11T22:50:43.761 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:50:43.862 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:50:43.862 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:50:43.863 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:50:43.863 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:50:43.863 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:50:43.864 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:50:43.864 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:50:43.864 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:50:44.308 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:50:44.308 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:50:44.309 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:50:44.309 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:50:44.786 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:50:44.786 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:50:44.787 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:50:44.787 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:50:44.787 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:50:44.788 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:50:44.789 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:50:44.789 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:50:44.789 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:50:45.314 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=21841308 2020-03-11T22:50:45.314 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 21841308 '!=' null 2020-03-11T22:50:45.314 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:50:45.314 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:45.314 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:50:45.315 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:45.405 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2020-03-11T22:50:45.405 INFO:tasks.workunit.client.0.smithi059.stdout: "local_reservations": { 2020-03-11T22:50:45.406 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:50:45.406 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:50:45.406 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [], 2020-03-11T22:50:45.406 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [ 2020-03-11T22:50:45.406 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:45.406 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "1.0", 2020-03-11T22:50:45.407 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 110, 2020-03-11T22:50:45.407 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:50:45.407 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:45.407 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:50:45.407 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:50:45.407 INFO:tasks.workunit.client.0.smithi059.stdout: "remote_reservations": { 2020-03-11T22:50:45.407 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:50:45.407 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:50:45.408 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [], 2020-03-11T22:50:45.408 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [] 2020-03-11T22:50:45.408 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:45.408 INFO:tasks.workunit.client.0.smithi059.stdout:} 2020-03-11T22:50:45.412 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:50:45.513 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:50:45.514 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:50:45.515 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:50:45.515 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:50:45.515 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:50:45.515 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:50:45.515 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:50:45.515 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:50:45.872 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:50:45.872 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:50:45.873 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:50:45.873 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:50:46.351 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:50:46.351 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:50:46.352 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:50:46.352 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:50:46.352 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:50:46.352 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:50:46.352 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:50:46.352 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:50:46.353 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:50:46.913 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=17036382 2020-03-11T22:50:46.913 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 17036382 '!=' null 2020-03-11T22:50:46.913 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:50:46.913 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:46.913 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:50:46.914 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:47.005 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2020-03-11T22:50:47.006 INFO:tasks.workunit.client.0.smithi059.stdout: "local_reservations": { 2020-03-11T22:50:47.006 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:50:47.006 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:50:47.006 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [], 2020-03-11T22:50:47.006 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [ 2020-03-11T22:50:47.006 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:47.006 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "1.0", 2020-03-11T22:50:47.007 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 110, 2020-03-11T22:50:47.007 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:50:47.007 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:47.007 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:50:47.007 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:50:47.007 INFO:tasks.workunit.client.0.smithi059.stdout: "remote_reservations": { 2020-03-11T22:50:47.007 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:50:47.007 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:50:47.008 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [], 2020-03-11T22:50:47.008 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [] 2020-03-11T22:50:47.008 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:47.008 INFO:tasks.workunit.client.0.smithi059.stdout:} 2020-03-11T22:50:47.013 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:50:47.114 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:50:47.114 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:50:47.115 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:50:47.116 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:50:47.116 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:50:47.116 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:50:47.116 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:50:47.116 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:50:47.477 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:50:47.477 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:50:47.477 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:50:47.477 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:50:47.959 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:50:47.959 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 1 2020-03-11T22:50:47.959 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:50:47.959 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=2 2020-03-11T22:50:47.960 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:47.960 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:50:47.960 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:48.051 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2020-03-11T22:50:48.051 INFO:tasks.workunit.client.0.smithi059.stdout: "local_reservations": { 2020-03-11T22:50:48.052 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:50:48.052 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:50:48.052 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [], 2020-03-11T22:50:48.052 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [ 2020-03-11T22:50:48.052 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:48.052 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "1.0", 2020-03-11T22:50:48.053 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 110, 2020-03-11T22:50:48.053 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:50:48.053 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:48.053 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:50:48.053 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:50:48.053 INFO:tasks.workunit.client.0.smithi059.stdout: "remote_reservations": { 2020-03-11T22:50:48.054 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:50:48.054 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:50:48.054 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [], 2020-03-11T22:50:48.054 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [] 2020-03-11T22:50:48.054 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:48.054 INFO:tasks.workunit.client.0.smithi059.stdout:} 2020-03-11T22:50:48.059 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:50:48.160 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:50:48.160 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:50:48.161 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:50:48.161 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:50:48.161 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:50:48.161 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:50:48.162 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:50:48.162 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:50:48.517 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:50:48.517 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:50:48.518 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:50:48.518 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:50:49.004 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:50:49.005 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:50:49.005 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:50:49.005 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:50:49.005 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:50:49.005 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:50:49.006 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:50:49.006 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:50:49.006 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:50:49.546 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=13104889 2020-03-11T22:50:49.546 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 13104889 '!=' null 2020-03-11T22:50:49.546 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:50:49.547 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:49.547 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:50:49.547 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:49.640 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2020-03-11T22:50:49.640 INFO:tasks.workunit.client.0.smithi059.stdout: "local_reservations": { 2020-03-11T22:50:49.641 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:50:49.641 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:50:49.641 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [], 2020-03-11T22:50:49.641 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [ 2020-03-11T22:50:49.641 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:49.641 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "1.0", 2020-03-11T22:50:49.641 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 110, 2020-03-11T22:50:49.642 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:50:49.642 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:49.642 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:50:49.642 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:50:49.642 INFO:tasks.workunit.client.0.smithi059.stdout: "remote_reservations": { 2020-03-11T22:50:49.642 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:50:49.642 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:50:49.643 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [], 2020-03-11T22:50:49.643 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [] 2020-03-11T22:50:49.643 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:49.643 INFO:tasks.workunit.client.0.smithi059.stdout:} 2020-03-11T22:50:49.647 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:50:49.748 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:50:49.748 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:50:49.749 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:50:49.750 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:50:49.750 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:50:49.750 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:50:49.750 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:50:49.750 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:50:50.253 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:50:50.254 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:50:50.254 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:50:50.254 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:50:50.731 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:50:50.732 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:50:50.732 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:50:50.732 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:50:50.732 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:50:50.732 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:50:50.732 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:50:50.732 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:50:50.733 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:50:51.213 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=13104889 2020-03-11T22:50:51.214 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 13104889 '!=' null 2020-03-11T22:50:51.214 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:50:51.214 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:51.214 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:50:51.214 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:51.308 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2020-03-11T22:50:51.308 INFO:tasks.workunit.client.0.smithi059.stdout: "local_reservations": { 2020-03-11T22:50:51.308 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:50:51.308 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:50:51.308 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [], 2020-03-11T22:50:51.309 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [ 2020-03-11T22:50:51.309 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:51.309 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "1.0", 2020-03-11T22:50:51.309 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 110, 2020-03-11T22:50:51.309 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:50:51.309 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:51.309 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:50:51.310 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:50:51.310 INFO:tasks.workunit.client.0.smithi059.stdout: "remote_reservations": { 2020-03-11T22:50:51.310 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:50:51.310 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:50:51.310 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [], 2020-03-11T22:50:51.310 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [] 2020-03-11T22:50:51.310 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:51.311 INFO:tasks.workunit.client.0.smithi059.stdout:} 2020-03-11T22:50:51.316 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:50:51.417 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:50:51.417 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:50:51.418 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:50:51.418 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:50:51.419 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:50:51.419 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:50:51.419 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:50:51.419 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:50:51.804 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:50:51.805 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:50:51.805 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:50:51.805 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:50:52.284 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:50:52.285 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:50:52.285 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:50:52.285 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:50:52.285 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:50:52.285 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:50:52.286 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:50:52.286 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:50:52.286 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:50:52.771 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=27520433 2020-03-11T22:50:52.771 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 27520433 '!=' null 2020-03-11T22:50:52.772 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:50:52.772 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:52.772 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:50:52.772 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:52.863 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2020-03-11T22:50:52.863 INFO:tasks.workunit.client.0.smithi059.stdout: "local_reservations": { 2020-03-11T22:50:52.863 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:50:52.863 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:50:52.863 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [], 2020-03-11T22:50:52.864 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [ 2020-03-11T22:50:52.864 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:52.864 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "1.0", 2020-03-11T22:50:52.864 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 110, 2020-03-11T22:50:52.865 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:50:52.865 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:52.865 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:50:52.865 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:50:52.865 INFO:tasks.workunit.client.0.smithi059.stdout: "remote_reservations": { 2020-03-11T22:50:52.866 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:50:52.866 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:50:52.866 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [], 2020-03-11T22:50:52.866 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [] 2020-03-11T22:50:52.867 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:52.867 INFO:tasks.workunit.client.0.smithi059.stdout:} 2020-03-11T22:50:52.870 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:50:52.971 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:50:52.971 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:50:52.973 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:50:52.973 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:50:52.973 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:50:52.974 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:50:52.974 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:50:52.974 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:50:53.329 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:50:53.330 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:50:53.330 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:50:53.330 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:50:53.834 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:50:53.834 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:50:53.835 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:50:53.835 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:50:53.835 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:50:53.835 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:50:53.835 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:50:53.835 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:50:53.835 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:50:54.378 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=22021554 2020-03-11T22:50:54.379 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 22021554 '!=' null 2020-03-11T22:50:54.379 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:50:54.379 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:54.379 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:50:54.379 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:54.471 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2020-03-11T22:50:54.471 INFO:tasks.workunit.client.0.smithi059.stdout: "local_reservations": { 2020-03-11T22:50:54.472 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:50:54.472 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:50:54.472 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [], 2020-03-11T22:50:54.472 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [ 2020-03-11T22:50:54.472 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:54.472 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "1.0", 2020-03-11T22:50:54.472 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 110, 2020-03-11T22:50:54.473 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:50:54.473 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:54.473 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:50:54.473 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:50:54.473 INFO:tasks.workunit.client.0.smithi059.stdout: "remote_reservations": { 2020-03-11T22:50:54.473 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:50:54.474 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:50:54.474 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [], 2020-03-11T22:50:54.474 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [] 2020-03-11T22:50:54.474 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:54.474 INFO:tasks.workunit.client.0.smithi059.stdout:} 2020-03-11T22:50:54.479 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:50:54.580 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:50:54.580 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:50:54.581 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:50:54.581 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:50:54.581 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:50:54.581 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:50:54.581 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:50:54.582 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:50:54.939 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:50:54.940 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:50:54.940 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:50:54.940 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:50:55.422 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:50:55.422 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:50:55.422 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:50:55.422 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:50:55.422 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:50:55.422 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:50:55.423 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:50:55.423 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:50:55.423 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:50:55.897 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=19235704 2020-03-11T22:50:55.897 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 19235704 '!=' null 2020-03-11T22:50:55.898 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:50:55.898 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:55.898 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:50:55.898 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:55.990 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2020-03-11T22:50:55.991 INFO:tasks.workunit.client.0.smithi059.stdout: "local_reservations": { 2020-03-11T22:50:55.991 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:50:55.991 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:50:55.991 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [], 2020-03-11T22:50:55.991 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [ 2020-03-11T22:50:55.991 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:55.991 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "1.0", 2020-03-11T22:50:55.992 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 110, 2020-03-11T22:50:55.992 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:50:55.992 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:55.992 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:50:55.992 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:50:55.992 INFO:tasks.workunit.client.0.smithi059.stdout: "remote_reservations": { 2020-03-11T22:50:55.992 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:50:55.993 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:50:55.993 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [], 2020-03-11T22:50:55.993 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [] 2020-03-11T22:50:55.993 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:55.993 INFO:tasks.workunit.client.0.smithi059.stdout:} 2020-03-11T22:50:55.998 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:50:56.099 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:50:56.099 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:50:56.100 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:50:56.100 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:50:56.101 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:50:56.101 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:50:56.101 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:50:56.101 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:50:56.458 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:50:56.461 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:50:56.461 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:50:56.461 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:50:56.936 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:50:56.937 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:50:56.937 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:50:56.937 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:50:56.937 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:50:56.937 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:50:56.937 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:50:56.937 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:50:56.938 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:50:57.457 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=28831205 2020-03-11T22:50:57.457 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 28831205 '!=' null 2020-03-11T22:50:57.457 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:50:57.457 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:57.457 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:50:57.457 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:57.549 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2020-03-11T22:50:57.549 INFO:tasks.workunit.client.0.smithi059.stdout: "local_reservations": { 2020-03-11T22:50:57.549 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:50:57.549 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:50:57.550 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [], 2020-03-11T22:50:57.550 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [ 2020-03-11T22:50:57.550 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:57.550 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "1.0", 2020-03-11T22:50:57.550 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 110, 2020-03-11T22:50:57.550 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:50:57.550 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:57.551 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:50:57.551 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:50:57.551 INFO:tasks.workunit.client.0.smithi059.stdout: "remote_reservations": { 2020-03-11T22:50:57.551 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:50:57.551 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:50:57.551 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [], 2020-03-11T22:50:57.551 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [] 2020-03-11T22:50:57.551 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:57.552 INFO:tasks.workunit.client.0.smithi059.stdout:} 2020-03-11T22:50:57.556 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:50:57.658 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:50:57.658 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:50:57.658 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:50:57.658 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:50:57.658 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:50:57.658 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:50:57.659 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:50:57.659 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:50:58.019 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:50:58.020 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:50:58.020 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:50:58.020 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:50:58.499 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:50:58.499 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:50:58.499 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:50:58.499 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:50:58.499 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:50:58.500 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:50:58.500 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:50:58.500 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:50:58.500 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:50:58.976 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=28831205 2020-03-11T22:50:58.976 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 28831205 '!=' null 2020-03-11T22:50:58.976 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:50:58.976 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:58.976 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:50:58.976 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:59.068 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2020-03-11T22:50:59.068 INFO:tasks.workunit.client.0.smithi059.stdout: "local_reservations": { 2020-03-11T22:50:59.068 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:50:59.068 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:50:59.069 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [], 2020-03-11T22:50:59.069 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [ 2020-03-11T22:50:59.069 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:50:59.069 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "1.0", 2020-03-11T22:50:59.069 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 110, 2020-03-11T22:50:59.069 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:50:59.070 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:59.070 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:50:59.070 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:50:59.070 INFO:tasks.workunit.client.0.smithi059.stdout: "remote_reservations": { 2020-03-11T22:50:59.070 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:50:59.070 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:50:59.070 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [], 2020-03-11T22:50:59.071 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [] 2020-03-11T22:50:59.071 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:50:59.071 INFO:tasks.workunit.client.0.smithi059.stdout:} 2020-03-11T22:50:59.075 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:50:59.177 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:50:59.177 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:50:59.178 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:50:59.178 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:50:59.178 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:50:59.178 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:50:59.178 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:50:59.178 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:50:59.534 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:50:59.535 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:50:59.535 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:50:59.535 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:00.021 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:51:00.021 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:51:00.021 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:00.022 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:00.022 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:00.022 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:00.022 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:00.022 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:00.022 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:00.584 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=20094674 2020-03-11T22:51:00.584 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 20094674 '!=' null 2020-03-11T22:51:00.584 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:00.584 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:00.584 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:00.585 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:00.676 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2020-03-11T22:51:00.676 INFO:tasks.workunit.client.0.smithi059.stdout: "local_reservations": { 2020-03-11T22:51:00.676 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:51:00.676 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:51:00.676 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [], 2020-03-11T22:51:00.677 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [ 2020-03-11T22:51:00.677 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:00.677 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "1.0", 2020-03-11T22:51:00.677 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 110, 2020-03-11T22:51:00.678 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:51:00.678 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:51:00.678 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:51:00.678 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:00.679 INFO:tasks.workunit.client.0.smithi059.stdout: "remote_reservations": { 2020-03-11T22:51:00.679 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:51:00.679 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:51:00.679 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [], 2020-03-11T22:51:00.680 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [] 2020-03-11T22:51:00.680 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:51:00.680 INFO:tasks.workunit.client.0.smithi059.stdout:} 2020-03-11T22:51:00.683 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:00.784 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:00.785 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:00.786 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:00.786 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:00.786 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:00.786 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:00.786 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:00.786 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:01.146 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:51:01.147 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:01.147 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:01.148 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:01.636 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:51:01.636 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:51:01.637 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:01.637 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:01.637 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:01.637 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:01.637 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:01.637 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:01.638 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:02.117 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=30578776 2020-03-11T22:51:02.117 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 30578776 '!=' null 2020-03-11T22:51:02.117 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:02.118 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:02.118 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:02.118 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:02.211 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2020-03-11T22:51:02.211 INFO:tasks.workunit.client.0.smithi059.stdout: "local_reservations": { 2020-03-11T22:51:02.211 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:51:02.211 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:51:02.211 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [], 2020-03-11T22:51:02.212 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [ 2020-03-11T22:51:02.212 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:02.212 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "1.0", 2020-03-11T22:51:02.212 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 110, 2020-03-11T22:51:02.213 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:51:02.213 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:51:02.213 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:51:02.213 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:02.213 INFO:tasks.workunit.client.0.smithi059.stdout: "remote_reservations": { 2020-03-11T22:51:02.214 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:51:02.214 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:51:02.214 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [], 2020-03-11T22:51:02.214 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [] 2020-03-11T22:51:02.215 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:51:02.215 INFO:tasks.workunit.client.0.smithi059.stdout:} 2020-03-11T22:51:02.220 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:02.321 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:02.321 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:02.323 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:02.323 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:02.323 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:02.323 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:02.324 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:02.324 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:02.687 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:51:02.687 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:02.688 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:02.688 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:03.168 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:51:03.168 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:51:03.168 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:03.169 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:03.169 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:03.169 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:03.169 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:03.169 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:03.169 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:03.658 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=20968152 2020-03-11T22:51:03.659 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 20968152 '!=' null 2020-03-11T22:51:03.659 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:03.659 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:03.659 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:03.659 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:03.751 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2020-03-11T22:51:03.751 INFO:tasks.workunit.client.0.smithi059.stdout: "local_reservations": { 2020-03-11T22:51:03.751 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:51:03.752 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:51:03.752 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [], 2020-03-11T22:51:03.752 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [ 2020-03-11T22:51:03.752 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:03.752 INFO:tasks.workunit.client.0.smithi059.stdout: "item": "1.0", 2020-03-11T22:51:03.752 INFO:tasks.workunit.client.0.smithi059.stdout: "prio": 110, 2020-03-11T22:51:03.752 INFO:tasks.workunit.client.0.smithi059.stdout: "can_preempt": true 2020-03-11T22:51:03.752 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:51:03.753 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:51:03.753 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:03.753 INFO:tasks.workunit.client.0.smithi059.stdout: "remote_reservations": { 2020-03-11T22:51:03.753 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:51:03.753 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:51:03.753 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [], 2020-03-11T22:51:03.753 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [] 2020-03-11T22:51:03.754 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:51:03.754 INFO:tasks.workunit.client.0.smithi059.stdout:} 2020-03-11T22:51:03.758 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:03.859 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:03.859 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:03.860 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:03.861 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:03.861 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:03.861 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:03.861 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:03.861 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:04.256 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:51:04.257 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:04.257 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:04.257 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:04.733 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:51:04.733 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:51:04.733 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:04.733 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:04.734 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:04.734 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:04.734 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:04.734 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:04.734 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:05.212 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=20968152 2020-03-11T22:51:05.212 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 20968152 '!=' null 2020-03-11T22:51:05.212 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:05.212 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:05.213 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:05.213 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:05.305 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2020-03-11T22:51:05.305 INFO:tasks.workunit.client.0.smithi059.stdout: "local_reservations": { 2020-03-11T22:51:05.306 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:51:05.306 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:51:05.306 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [], 2020-03-11T22:51:05.306 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [] 2020-03-11T22:51:05.306 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:05.306 INFO:tasks.workunit.client.0.smithi059.stdout: "remote_reservations": { 2020-03-11T22:51:05.306 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:51:05.307 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:51:05.307 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [], 2020-03-11T22:51:05.307 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [] 2020-03-11T22:51:05.307 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:51:05.307 INFO:tasks.workunit.client.0.smithi059.stdout:} 2020-03-11T22:51:05.312 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:05.414 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:05.414 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:05.415 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:05.415 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:05.415 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:05.415 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:05.415 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:05.415 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:05.773 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:51:05.773 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:05.774 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:05.774 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:06.262 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:51:06.262 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:51:06.262 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:06.262 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:06.263 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:06.263 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:06.263 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:06.263 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:06.263 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:06.740 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=20968126 2020-03-11T22:51:06.740 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 20968126 '!=' null 2020-03-11T22:51:06.741 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:06.741 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:06.741 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:51:06.741 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:06.832 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2020-03-11T22:51:06.832 INFO:tasks.workunit.client.0.smithi059.stdout: "local_reservations": { 2020-03-11T22:51:06.832 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:51:06.833 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:51:06.833 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [], 2020-03-11T22:51:06.833 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [] 2020-03-11T22:51:06.833 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:06.833 INFO:tasks.workunit.client.0.smithi059.stdout: "remote_reservations": { 2020-03-11T22:51:06.833 INFO:tasks.workunit.client.0.smithi059.stdout: "max_allowed": 1, 2020-03-11T22:51:06.833 INFO:tasks.workunit.client.0.smithi059.stdout: "min_priority": 0, 2020-03-11T22:51:06.834 INFO:tasks.workunit.client.0.smithi059.stdout: "queues": [], 2020-03-11T22:51:06.834 INFO:tasks.workunit.client.0.smithi059.stdout: "in_progress": [] 2020-03-11T22:51:06.834 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:51:06.834 INFO:tasks.workunit.client.0.smithi059.stdout:} 2020-03-11T22:51:06.839 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:06.941 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:06.941 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:06.942 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:06.942 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:06.942 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:06.942 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:06.942 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:06.942 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:07.298 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=3 2020-03-11T22:51:07.299 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:07.299 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:07.299 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:07.775 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 3 = 3 2020-03-11T22:51:07.775 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:51:07.775 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:51:07.775 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:298: TEST_backfill_priority: ceph pg dump pgs 2020-03-11T22:51:08.120 INFO:tasks.workunit.client.0.smithi059.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:51:08.121 INFO:tasks.workunit.client.0.smithi059.stdout:7.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2020-03-11T22:50:22.784188+0000 97'150 121:343 [1,4] 1 [1,4] 1 0'0 2020-03-11T22:47:44.646052+0000 0'0 2020-03-11T22:47:44.646052+0000 0 2020-03-11T22:51:08.121 INFO:tasks.workunit.client.0.smithi059.stdout:1.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2020-03-11T22:51:05.660319+0000 97'150 122:360 [1,2] 1 [1,2] 1 0'0 2020-03-11T22:47:30.333509+0000 0'0 2020-03-11T22:47:30.333509+0000 0 2020-03-11T22:51:08.121 INFO:tasks.workunit.client.0.smithi059.stdout:3.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2020-03-11T22:50:43.833475+0000 97'150 121:364 [1,2] 1 [1,2] 1 0'0 2020-03-11T22:47:35.075766+0000 0'0 2020-03-11T22:47:35.075766+0000 0 2020-03-11T22:51:08.121 INFO:tasks.workunit.client.0.smithi059.stdout: 2020-03-11T22:51:08.121 INFO:tasks.workunit.client.0.smithi059.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:51:08.122 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2020-03-11T22:51:08.131 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:300: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:51:08.131 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:51:08.132 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:51:08.132 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:51:08.132 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:51:08.132 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56462 2020-03-11T22:51:08.132 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56462/ceph-osd.1.asok 2020-03-11T22:51:08.133 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:300: TEST_backfill_priority: CEPH_ARGS= 2020-03-11T22:51:08.133 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:300: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.56462/ceph-osd.1.asok dump_pgstate_history 2020-03-11T22:51:08.225 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2020-03-11T22:51:08.225 INFO:tasks.workunit.client.0.smithi059.stdout: "pgs": [ 2020-03-11T22:51:08.226 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.226 INFO:tasks.workunit.client.0.smithi059.stdout: "pg": "7.0", 2020-03-11T22:51:08.226 INFO:tasks.workunit.client.0.smithi059.stdout: "currently": "Started/Primary/Active/Clean", 2020-03-11T22:51:08.226 INFO:tasks.workunit.client.0.smithi059.stdout: "history": [ 2020-03-11T22:51:08.226 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.226 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": "63", 2020-03-11T22:51:08.227 INFO:tasks.workunit.client.0.smithi059.stdout: "states": [ 2020-03-11T22:51:08.227 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.227 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Start", 2020-03-11T22:51:08.227 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:45.480275+0000", 2020-03-11T22:51:08.227 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:47:45.480286+0000" 2020-03-11T22:51:08.227 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.227 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.228 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:51:08.228 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:45.480295+0000", 2020-03-11T22:51:08.228 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:47:45.481014+0000" 2020-03-11T22:51:08.228 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.228 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.228 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:51:08.229 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:45.481015+0000", 2020-03-11T22:51:08.229 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:47:45.481133+0000" 2020-03-11T22:51:08.229 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.229 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.229 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:51:08.229 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:45.481134+0000", 2020-03-11T22:51:08.229 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:47:45.481161+0000" 2020-03-11T22:51:08.230 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.230 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.230 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:51:08.230 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:45.481161+0000", 2020-03-11T22:51:08.230 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:47:45.693021+0000" 2020-03-11T22:51:08.230 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.230 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.231 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:51:08.231 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:45.480291+0000", 2020-03-11T22:51:08.231 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:47:45.693030+0000" 2020-03-11T22:51:08.231 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.231 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.231 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:51:08.231 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:45.693224+0000", 2020-03-11T22:51:08.232 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:47:46.367670+0000" 2020-03-11T22:51:08.232 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.232 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.232 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Active", 2020-03-11T22:51:08.232 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:45.693030+0000", 2020-03-11T22:51:08.232 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:47:46.367692+0000" 2020-03-11T22:51:08.232 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.232 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.233 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary", 2020-03-11T22:51:08.233 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:45.480286+0000", 2020-03-11T22:51:08.233 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:47:46.367707+0000" 2020-03-11T22:51:08.233 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.233 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.233 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started", 2020-03-11T22:51:08.233 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:45.480270+0000", 2020-03-11T22:51:08.234 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:47:46.367712+0000" 2020-03-11T22:51:08.234 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:51:08.234 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:51:08.234 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.234 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.234 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": "63", 2020-03-11T22:51:08.234 INFO:tasks.workunit.client.0.smithi059.stdout: "states": [ 2020-03-11T22:51:08.234 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.235 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Reset", 2020-03-11T22:51:08.235 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:46.367713+0000", 2020-03-11T22:51:08.235 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:47:46.367907+0000" 2020-03-11T22:51:08.235 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:51:08.235 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:51:08.235 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.235 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.236 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": "95", 2020-03-11T22:51:08.236 INFO:tasks.workunit.client.0.smithi059.stdout: "states": [ 2020-03-11T22:51:08.236 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.236 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Start", 2020-03-11T22:51:08.236 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:46.367918+0000", 2020-03-11T22:51:08.236 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:47:46.367928+0000" 2020-03-11T22:51:08.237 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.237 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.237 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:51:08.237 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:46.367936+0000", 2020-03-11T22:51:08.237 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:47:46.528577+0000" 2020-03-11T22:51:08.238 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.238 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.238 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:51:08.238 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:46.528577+0000", 2020-03-11T22:51:08.238 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:47:46.528648+0000" 2020-03-11T22:51:08.239 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.239 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.239 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:51:08.239 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:46.528649+0000", 2020-03-11T22:51:08.239 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:47:46.528662+0000" 2020-03-11T22:51:08.240 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.240 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.240 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:51:08.240 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:46.528662+0000", 2020-03-11T22:51:08.240 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:47:46.666298+0000" 2020-03-11T22:51:08.241 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.241 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.241 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:51:08.241 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:46.367933+0000", 2020-03-11T22:51:08.241 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:47:46.666309+0000" 2020-03-11T22:51:08.241 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.241 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.242 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:51:08.242 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:46.666428+0000", 2020-03-11T22:51:08.242 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:47:47.427544+0000" 2020-03-11T22:51:08.242 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.242 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.242 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Active/Recovered", 2020-03-11T22:51:08.242 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:47.427545+0000", 2020-03-11T22:51:08.243 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:47:47.427590+0000" 2020-03-11T22:51:08.243 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.243 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.243 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Active/Clean", 2020-03-11T22:51:08.243 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:47.427591+0000", 2020-03-11T22:51:08.243 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:48:14.319167+0000" 2020-03-11T22:51:08.244 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.244 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.244 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Active", 2020-03-11T22:51:08.244 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:46.666309+0000", 2020-03-11T22:51:08.244 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:48:14.319200+0000" 2020-03-11T22:51:08.244 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.244 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.245 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary", 2020-03-11T22:51:08.245 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:46.367928+0000", 2020-03-11T22:51:08.245 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:48:14.319231+0000" 2020-03-11T22:51:08.245 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.245 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.245 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started", 2020-03-11T22:51:08.245 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:46.367907+0000", 2020-03-11T22:51:08.245 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:48:14.319243+0000" 2020-03-11T22:51:08.246 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:51:08.246 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:51:08.246 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.246 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.246 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": "96", 2020-03-11T22:51:08.246 INFO:tasks.workunit.client.0.smithi059.stdout: "states": [ 2020-03-11T22:51:08.246 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.247 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Reset", 2020-03-11T22:51:08.247 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:48:14.319244+0000", 2020-03-11T22:51:08.247 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:48:14.319644+0000" 2020-03-11T22:51:08.247 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:51:08.247 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:51:08.247 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.247 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.248 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": "102", 2020-03-11T22:51:08.248 INFO:tasks.workunit.client.0.smithi059.stdout: "states": [ 2020-03-11T22:51:08.248 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.248 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Start", 2020-03-11T22:51:08.248 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:48:14.319653+0000", 2020-03-11T22:51:08.248 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:48:14.319677+0000" 2020-03-11T22:51:08.248 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.248 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.249 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:51:08.249 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:48:14.319691+0000", 2020-03-11T22:51:08.249 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:48:14.319900+0000" 2020-03-11T22:51:08.249 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.249 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.249 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:51:08.249 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:48:14.319901+0000", 2020-03-11T22:51:08.250 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:48:14.320009+0000" 2020-03-11T22:51:08.250 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.250 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.250 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:51:08.250 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:48:14.320010+0000", 2020-03-11T22:51:08.250 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:48:14.320037+0000" 2020-03-11T22:51:08.251 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.251 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.251 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:51:08.251 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:48:14.320037+0000", 2020-03-11T22:51:08.251 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:48:14.492537+0000" 2020-03-11T22:51:08.251 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.251 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.252 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:51:08.252 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:48:14.319685+0000", 2020-03-11T22:51:08.252 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:48:14.492554+0000" 2020-03-11T22:51:08.252 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.252 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.252 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:51:08.252 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:48:14.492697+0000", 2020-03-11T22:51:08.252 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:48:14.500863+0000" 2020-03-11T22:51:08.253 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.253 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.253 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Active/Recovered", 2020-03-11T22:51:08.253 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:48:14.500863+0000", 2020-03-11T22:51:08.253 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:48:14.500914+0000" 2020-03-11T22:51:08.253 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.253 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.254 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Active/Clean", 2020-03-11T22:51:08.254 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:48:14.500914+0000", 2020-03-11T22:51:08.254 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:49:14.887783+0000" 2020-03-11T22:51:08.254 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.254 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.254 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Active", 2020-03-11T22:51:08.254 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:48:14.492554+0000", 2020-03-11T22:51:08.255 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:49:14.887803+0000" 2020-03-11T22:51:08.255 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.255 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.255 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary", 2020-03-11T22:51:08.255 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:48:14.319678+0000", 2020-03-11T22:51:08.255 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:49:14.887845+0000" 2020-03-11T22:51:08.255 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.256 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.256 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started", 2020-03-11T22:51:08.256 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:48:14.319644+0000", 2020-03-11T22:51:08.256 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:49:14.887850+0000" 2020-03-11T22:51:08.256 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:51:08.256 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:51:08.256 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.256 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.257 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": "102", 2020-03-11T22:51:08.257 INFO:tasks.workunit.client.0.smithi059.stdout: "states": [ 2020-03-11T22:51:08.257 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.257 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Reset", 2020-03-11T22:51:08.257 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:49:14.887850+0000", 2020-03-11T22:51:08.257 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:49:14.888124+0000" 2020-03-11T22:51:08.257 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:51:08.258 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:51:08.258 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.258 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.258 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": "103", 2020-03-11T22:51:08.258 INFO:tasks.workunit.client.0.smithi059.stdout: "states": [ 2020-03-11T22:51:08.259 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.259 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Start", 2020-03-11T22:51:08.259 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:49:14.888128+0000", 2020-03-11T22:51:08.259 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:49:14.888140+0000" 2020-03-11T22:51:08.259 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.260 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.260 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:51:08.260 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:49:14.888147+0000", 2020-03-11T22:51:08.260 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:49:14.931212+0000" 2020-03-11T22:51:08.260 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.260 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.260 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:51:08.261 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:49:14.931212+0000", 2020-03-11T22:51:08.261 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:49:14.931264+0000" 2020-03-11T22:51:08.261 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.261 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.261 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:51:08.261 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:49:14.888143+0000", 2020-03-11T22:51:08.261 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:49:14.931275+0000" 2020-03-11T22:51:08.261 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.262 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.262 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/WaitActingChange", 2020-03-11T22:51:08.262 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:49:14.931275+0000", 2020-03-11T22:51:08.262 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:49:15.064833+0000" 2020-03-11T22:51:08.262 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.263 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.263 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary", 2020-03-11T22:51:08.263 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:49:14.888140+0000", 2020-03-11T22:51:08.263 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:49:15.064852+0000" 2020-03-11T22:51:08.263 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.263 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.263 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started", 2020-03-11T22:51:08.264 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:49:14.888124+0000", 2020-03-11T22:51:08.264 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:49:15.064856+0000" 2020-03-11T22:51:08.264 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:51:08.264 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:51:08.265 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.265 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.265 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": "103", 2020-03-11T22:51:08.265 INFO:tasks.workunit.client.0.smithi059.stdout: "states": [ 2020-03-11T22:51:08.266 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.266 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Reset", 2020-03-11T22:51:08.266 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:49:15.064856+0000", 2020-03-11T22:51:08.266 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:49:15.065055+0000" 2020-03-11T22:51:08.266 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:51:08.267 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:51:08.267 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.267 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.267 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": "117", 2020-03-11T22:51:08.267 INFO:tasks.workunit.client.0.smithi059.stdout: "states": [ 2020-03-11T22:51:08.267 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.268 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Start", 2020-03-11T22:51:08.268 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:49:15.065059+0000", 2020-03-11T22:51:08.268 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:49:15.065071+0000" 2020-03-11T22:51:08.268 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.268 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.268 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:51:08.268 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:49:15.065078+0000", 2020-03-11T22:51:08.269 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:49:15.156955+0000" 2020-03-11T22:51:08.269 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.269 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.269 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:51:08.269 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:49:15.156956+0000", 2020-03-11T22:51:08.269 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:49:15.157031+0000" 2020-03-11T22:51:08.270 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.270 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.270 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:51:08.270 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:49:15.157031+0000", 2020-03-11T22:51:08.270 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:49:15.157049+0000" 2020-03-11T22:51:08.270 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.270 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.271 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:51:08.271 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:49:15.157049+0000", 2020-03-11T22:51:08.271 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:49:15.366287+0000" 2020-03-11T22:51:08.271 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.271 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.271 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:51:08.272 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:49:15.065075+0000", 2020-03-11T22:51:08.272 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:49:15.366299+0000" 2020-03-11T22:51:08.272 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.272 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.272 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:51:08.272 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:49:15.366550+0000", 2020-03-11T22:51:08.272 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:49:15.453814+0000" 2020-03-11T22:51:08.273 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.273 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.273 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Active/WaitLocalBackfillReserved", 2020-03-11T22:51:08.273 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:49:15.453815+0000", 2020-03-11T22:51:08.273 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:49:15.454028+0000" 2020-03-11T22:51:08.273 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.273 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.274 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Active/WaitRemoteBackfillReserved", 2020-03-11T22:51:08.274 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:49:15.454029+0000", 2020-03-11T22:51:08.274 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:49:15.454535+0000" 2020-03-11T22:51:08.274 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.274 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.274 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Active/Backfilling", 2020-03-11T22:51:08.274 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:49:15.454536+0000", 2020-03-11T22:51:08.274 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:49:44.199796+0000" 2020-03-11T22:51:08.275 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.275 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.275 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Active/NotBackfilling", 2020-03-11T22:51:08.275 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:49:44.199797+0000", 2020-03-11T22:51:08.275 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:49:44.200156+0000" 2020-03-11T22:51:08.275 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.275 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.276 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Active/WaitLocalBackfillReserved", 2020-03-11T22:51:08.276 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:49:44.200157+0000", 2020-03-11T22:51:08.276 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:49:56.476862+0000" 2020-03-11T22:51:08.276 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.276 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.276 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Active/WaitRemoteBackfillReserved", 2020-03-11T22:51:08.277 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:49:56.476863+0000", 2020-03-11T22:51:08.277 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:49:56.477270+0000" 2020-03-11T22:51:08.277 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.277 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.277 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Active/Backfilling", 2020-03-11T22:51:08.278 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:49:56.477270+0000", 2020-03-11T22:51:08.278 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:50:22.053306+0000" 2020-03-11T22:51:08.278 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.278 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.278 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Active/Recovered", 2020-03-11T22:51:08.278 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:50:22.053307+0000", 2020-03-11T22:51:08.278 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:50:22.053447+0000" 2020-03-11T22:51:08.279 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.279 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.279 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Active/Clean", 2020-03-11T22:51:08.279 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:50:22.053448+0000", 2020-03-11T22:51:08.279 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:50:22.196834+0000" 2020-03-11T22:51:08.279 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.279 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.280 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Active", 2020-03-11T22:51:08.280 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:49:15.366300+0000", 2020-03-11T22:51:08.280 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:50:22.196865+0000" 2020-03-11T22:51:08.280 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.280 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.280 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary", 2020-03-11T22:51:08.280 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:49:15.065071+0000", 2020-03-11T22:51:08.281 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:50:22.196885+0000" 2020-03-11T22:51:08.281 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.281 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.281 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started", 2020-03-11T22:51:08.281 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:49:15.065055+0000", 2020-03-11T22:51:08.281 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:50:22.196892+0000" 2020-03-11T22:51:08.281 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:51:08.282 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:51:08.282 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.282 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.282 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": "117", 2020-03-11T22:51:08.282 INFO:tasks.workunit.client.0.smithi059.stdout: "states": [ 2020-03-11T22:51:08.282 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.282 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Reset", 2020-03-11T22:51:08.282 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:50:22.196892+0000", 2020-03-11T22:51:08.283 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:50:22.197172+0000" 2020-03-11T22:51:08.283 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:51:08.283 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:51:08.283 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:51:08.283 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:51:08.283 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.283 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.284 INFO:tasks.workunit.client.0.smithi059.stdout: "pg": "3.0", 2020-03-11T22:51:08.284 INFO:tasks.workunit.client.0.smithi059.stdout: "currently": "Started/Primary/Active/Clean", 2020-03-11T22:51:08.284 INFO:tasks.workunit.client.0.smithi059.stdout: "history": [ 2020-03-11T22:51:08.284 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.284 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": "41", 2020-03-11T22:51:08.284 INFO:tasks.workunit.client.0.smithi059.stdout: "states": [ 2020-03-11T22:51:08.284 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.285 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Start", 2020-03-11T22:51:08.285 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:35.304880+0000", 2020-03-11T22:51:08.285 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:47:35.304889+0000" 2020-03-11T22:51:08.285 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.285 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.285 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:51:08.285 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:35.304896+0000", 2020-03-11T22:51:08.286 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:47:35.392025+0000" 2020-03-11T22:51:08.286 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.286 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.286 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:51:08.286 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:35.392026+0000", 2020-03-11T22:51:08.286 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:47:35.392091+0000" 2020-03-11T22:51:08.287 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.287 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.287 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:51:08.287 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:35.392091+0000", 2020-03-11T22:51:08.287 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:47:35.392105+0000" 2020-03-11T22:51:08.287 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.288 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.288 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:51:08.288 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:35.392105+0000", 2020-03-11T22:51:08.288 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:47:35.581188+0000" 2020-03-11T22:51:08.288 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.288 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.289 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:51:08.289 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:35.304892+0000", 2020-03-11T22:51:08.289 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:47:35.581197+0000" 2020-03-11T22:51:08.289 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.289 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.290 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:51:08.290 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:35.581400+0000", 2020-03-11T22:51:08.290 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:47:36.294264+0000" 2020-03-11T22:51:08.290 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.290 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.290 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Active/Recovered", 2020-03-11T22:51:08.290 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:36.294265+0000", 2020-03-11T22:51:08.291 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:47:36.294347+0000" 2020-03-11T22:51:08.291 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.291 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.291 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Active/Clean", 2020-03-11T22:51:08.291 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:36.294347+0000", 2020-03-11T22:51:08.291 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:47:37.225336+0000" 2020-03-11T22:51:08.291 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.292 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.292 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Active", 2020-03-11T22:51:08.292 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:35.581198+0000", 2020-03-11T22:51:08.292 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:47:37.225361+0000" 2020-03-11T22:51:08.292 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.292 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.292 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary", 2020-03-11T22:51:08.293 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:35.304889+0000", 2020-03-11T22:51:08.293 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:47:37.225377+0000" 2020-03-11T22:51:08.293 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.293 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.293 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started", 2020-03-11T22:51:08.293 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:35.304877+0000", 2020-03-11T22:51:08.293 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:47:37.225381+0000" 2020-03-11T22:51:08.294 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:51:08.294 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:51:08.294 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.294 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.294 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": "42", 2020-03-11T22:51:08.294 INFO:tasks.workunit.client.0.smithi059.stdout: "states": [ 2020-03-11T22:51:08.295 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.295 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Reset", 2020-03-11T22:51:08.295 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:37.225381+0000", 2020-03-11T22:51:08.295 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:47:37.225635+0000" 2020-03-11T22:51:08.295 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:51:08.295 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:51:08.298 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.298 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.298 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": "92", 2020-03-11T22:51:08.298 INFO:tasks.workunit.client.0.smithi059.stdout: "states": [ 2020-03-11T22:51:08.298 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.299 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Start", 2020-03-11T22:51:08.299 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:37.225641+0000", 2020-03-11T22:51:08.299 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:47:37.225655+0000" 2020-03-11T22:51:08.299 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.300 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.300 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:51:08.300 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:37.225662+0000", 2020-03-11T22:51:08.300 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:47:37.330359+0000" 2020-03-11T22:51:08.300 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.301 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.301 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:51:08.301 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:37.330360+0000", 2020-03-11T22:51:08.301 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:47:37.330416+0000" 2020-03-11T22:51:08.302 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.302 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.302 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:51:08.302 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:37.330416+0000", 2020-03-11T22:51:08.302 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:47:37.330428+0000" 2020-03-11T22:51:08.303 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.303 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.303 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:51:08.303 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:37.330429+0000", 2020-03-11T22:51:08.304 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:47:37.442026+0000" 2020-03-11T22:51:08.304 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.304 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.304 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:51:08.304 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:37.225659+0000", 2020-03-11T22:51:08.305 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:47:37.442035+0000" 2020-03-11T22:51:08.305 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.305 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.305 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:51:08.306 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:37.442168+0000", 2020-03-11T22:51:08.306 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:47:37.800945+0000" 2020-03-11T22:51:08.306 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.306 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.306 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Active/Recovered", 2020-03-11T22:51:08.307 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:37.800946+0000", 2020-03-11T22:51:08.307 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:47:37.800993+0000" 2020-03-11T22:51:08.307 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.307 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.307 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Active/Clean", 2020-03-11T22:51:08.308 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:37.800994+0000", 2020-03-11T22:51:08.308 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:48:12.442424+0000" 2020-03-11T22:51:08.308 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.308 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.309 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Active", 2020-03-11T22:51:08.309 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:37.442035+0000", 2020-03-11T22:51:08.309 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:48:12.442447+0000" 2020-03-11T22:51:08.309 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.309 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.310 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary", 2020-03-11T22:51:08.310 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:37.225656+0000", 2020-03-11T22:51:08.310 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:48:12.442463+0000" 2020-03-11T22:51:08.310 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.311 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.311 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started", 2020-03-11T22:51:08.311 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:37.225636+0000", 2020-03-11T22:51:08.311 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:48:12.442470+0000" 2020-03-11T22:51:08.311 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:51:08.312 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:51:08.312 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.312 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.312 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": "92", 2020-03-11T22:51:08.312 INFO:tasks.workunit.client.0.smithi059.stdout: "states": [ 2020-03-11T22:51:08.313 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.313 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Reset", 2020-03-11T22:51:08.313 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:48:12.442473+0000", 2020-03-11T22:51:08.313 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:48:12.442671+0000" 2020-03-11T22:51:08.314 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:51:08.314 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:51:08.314 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.314 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.314 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": "109", 2020-03-11T22:51:08.315 INFO:tasks.workunit.client.0.smithi059.stdout: "states": [ 2020-03-11T22:51:08.315 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.315 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Start", 2020-03-11T22:51:08.315 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:48:12.442675+0000", 2020-03-11T22:51:08.316 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:48:12.442687+0000" 2020-03-11T22:51:08.316 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.316 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.316 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:51:08.316 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:48:12.442694+0000", 2020-03-11T22:51:08.317 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:48:12.575998+0000" 2020-03-11T22:51:08.317 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.317 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.317 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:51:08.317 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:48:12.575999+0000", 2020-03-11T22:51:08.318 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:48:12.576065+0000" 2020-03-11T22:51:08.318 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.318 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.318 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:51:08.318 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:48:12.576066+0000", 2020-03-11T22:51:08.319 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:48:12.576078+0000" 2020-03-11T22:51:08.319 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.319 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.319 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:51:08.320 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:48:12.576078+0000", 2020-03-11T22:51:08.320 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:48:12.717749+0000" 2020-03-11T22:51:08.320 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.320 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.321 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:51:08.321 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:48:12.442691+0000", 2020-03-11T22:51:08.321 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:48:12.717758+0000" 2020-03-11T22:51:08.322 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.322 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.322 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:51:08.322 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:48:12.717828+0000", 2020-03-11T22:51:08.322 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:48:12.753793+0000" 2020-03-11T22:51:08.323 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.323 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.323 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Active/Recovered", 2020-03-11T22:51:08.323 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:48:12.753794+0000", 2020-03-11T22:51:08.323 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:48:12.753831+0000" 2020-03-11T22:51:08.324 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.324 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.324 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Active/Clean", 2020-03-11T22:51:08.324 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:48:12.753832+0000", 2020-03-11T22:51:08.324 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:49:30.244976+0000" 2020-03-11T22:51:08.325 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.325 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.325 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Active", 2020-03-11T22:51:08.325 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:48:12.717759+0000", 2020-03-11T22:51:08.325 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:49:30.245005+0000" 2020-03-11T22:51:08.325 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.326 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.326 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary", 2020-03-11T22:51:08.326 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:48:12.442687+0000", 2020-03-11T22:51:08.326 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:49:30.245049+0000" 2020-03-11T22:51:08.326 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.327 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.327 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started", 2020-03-11T22:51:08.327 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:48:12.442672+0000", 2020-03-11T22:51:08.327 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:49:30.245053+0000" 2020-03-11T22:51:08.327 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:51:08.328 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:51:08.328 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.328 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.328 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": "110", 2020-03-11T22:51:08.329 INFO:tasks.workunit.client.0.smithi059.stdout: "states": [ 2020-03-11T22:51:08.329 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.329 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Reset", 2020-03-11T22:51:08.329 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:49:30.245054+0000", 2020-03-11T22:51:08.330 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:49:30.245372+0000" 2020-03-11T22:51:08.330 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:51:08.330 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:51:08.330 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.331 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.331 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": "111", 2020-03-11T22:51:08.331 INFO:tasks.workunit.client.0.smithi059.stdout: "states": [ 2020-03-11T22:51:08.331 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.332 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Start", 2020-03-11T22:51:08.332 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:49:30.245379+0000", 2020-03-11T22:51:08.332 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:49:30.245393+0000" 2020-03-11T22:51:08.332 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.333 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.333 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:51:08.333 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:49:30.245402+0000", 2020-03-11T22:51:08.333 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:49:30.272074+0000" 2020-03-11T22:51:08.333 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.334 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.334 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:51:08.334 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:49:30.272074+0000", 2020-03-11T22:51:08.334 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:49:30.272136+0000" 2020-03-11T22:51:08.335 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.335 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.335 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:51:08.335 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:49:30.245399+0000", 2020-03-11T22:51:08.336 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:49:30.272144+0000" 2020-03-11T22:51:08.336 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.336 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.336 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/WaitActingChange", 2020-03-11T22:51:08.336 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:49:30.272144+0000", 2020-03-11T22:51:08.337 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:49:30.489571+0000" 2020-03-11T22:51:08.337 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.337 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.337 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary", 2020-03-11T22:51:08.338 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:49:30.245394+0000", 2020-03-11T22:51:08.338 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:49:30.489588+0000" 2020-03-11T22:51:08.338 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.338 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.338 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started", 2020-03-11T22:51:08.338 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:49:30.245373+0000", 2020-03-11T22:51:08.339 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:49:30.489593+0000" 2020-03-11T22:51:08.339 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:51:08.339 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:51:08.339 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.339 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.339 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": "111", 2020-03-11T22:51:08.340 INFO:tasks.workunit.client.0.smithi059.stdout: "states": [ 2020-03-11T22:51:08.340 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.340 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Reset", 2020-03-11T22:51:08.340 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:49:30.489593+0000", 2020-03-11T22:51:08.340 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:49:30.489972+0000" 2020-03-11T22:51:08.341 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:51:08.341 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:51:08.341 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.341 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.341 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": "119", 2020-03-11T22:51:08.341 INFO:tasks.workunit.client.0.smithi059.stdout: "states": [ 2020-03-11T22:51:08.342 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.342 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Start", 2020-03-11T22:51:08.342 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:49:30.489982+0000", 2020-03-11T22:51:08.342 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:49:30.490013+0000" 2020-03-11T22:51:08.342 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.342 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.342 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:51:08.343 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:49:30.490032+0000", 2020-03-11T22:51:08.343 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:49:30.490472+0000" 2020-03-11T22:51:08.343 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.343 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.343 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:51:08.343 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:49:30.490472+0000", 2020-03-11T22:51:08.344 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:49:30.490539+0000" 2020-03-11T22:51:08.344 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.344 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.344 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:51:08.344 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:49:30.490540+0000", 2020-03-11T22:51:08.344 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:49:30.490556+0000" 2020-03-11T22:51:08.344 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.345 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.345 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:51:08.345 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:49:30.490556+0000", 2020-03-11T22:51:08.345 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:49:30.857733+0000" 2020-03-11T22:51:08.345 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.345 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.345 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:51:08.346 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:49:30.490023+0000", 2020-03-11T22:51:08.346 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:49:30.857746+0000" 2020-03-11T22:51:08.346 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.346 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.346 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:51:08.346 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:49:30.858005+0000", 2020-03-11T22:51:08.346 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:49:30.924746+0000" 2020-03-11T22:51:08.347 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.347 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.347 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Active/WaitLocalBackfillReserved", 2020-03-11T22:51:08.347 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:49:30.924746+0000", 2020-03-11T22:51:08.347 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:49:44.199957+0000" 2020-03-11T22:51:08.347 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.347 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.348 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Active/WaitRemoteBackfillReserved", 2020-03-11T22:51:08.348 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:49:44.199958+0000", 2020-03-11T22:51:08.348 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:49:44.200440+0000" 2020-03-11T22:51:08.348 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.349 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.349 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Active/Backfilling", 2020-03-11T22:51:08.349 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:49:44.200441+0000", 2020-03-11T22:51:08.349 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:49:56.476734+0000" 2020-03-11T22:51:08.349 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.349 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.350 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Active/NotBackfilling", 2020-03-11T22:51:08.350 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:49:56.476734+0000", 2020-03-11T22:51:08.350 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:49:56.476981+0000" 2020-03-11T22:51:08.350 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.350 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.350 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Active/WaitLocalBackfillReserved", 2020-03-11T22:51:08.350 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:49:56.476982+0000", 2020-03-11T22:51:08.350 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:50:22.053635+0000" 2020-03-11T22:51:08.351 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.351 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.351 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Active/WaitRemoteBackfillReserved", 2020-03-11T22:51:08.351 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:50:22.053635+0000", 2020-03-11T22:51:08.351 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:50:22.054052+0000" 2020-03-11T22:51:08.351 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.351 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.352 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Active/Backfilling", 2020-03-11T22:51:08.352 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:50:22.054052+0000", 2020-03-11T22:51:08.352 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:50:43.100659+0000" 2020-03-11T22:51:08.352 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.352 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.352 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Active/Recovered", 2020-03-11T22:51:08.352 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:50:43.100660+0000", 2020-03-11T22:51:08.353 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:50:43.100902+0000" 2020-03-11T22:51:08.353 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.353 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.353 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Active/Clean", 2020-03-11T22:51:08.353 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:50:43.100903+0000", 2020-03-11T22:51:08.353 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:50:43.382900+0000" 2020-03-11T22:51:08.353 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.354 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.354 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Active", 2020-03-11T22:51:08.354 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:49:30.857747+0000", 2020-03-11T22:51:08.354 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:50:43.382944+0000" 2020-03-11T22:51:08.354 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.354 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.354 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary", 2020-03-11T22:51:08.355 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:49:30.490014+0000", 2020-03-11T22:51:08.355 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:50:43.382979+0000" 2020-03-11T22:51:08.355 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.355 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.355 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started", 2020-03-11T22:51:08.355 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:49:30.489973+0000", 2020-03-11T22:51:08.355 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:50:43.382990+0000" 2020-03-11T22:51:08.356 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:51:08.356 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:51:08.356 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.356 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.356 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": "119", 2020-03-11T22:51:08.356 INFO:tasks.workunit.client.0.smithi059.stdout: "states": [ 2020-03-11T22:51:08.356 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.356 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Reset", 2020-03-11T22:51:08.357 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:50:43.382991+0000", 2020-03-11T22:51:08.357 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:50:43.383506+0000" 2020-03-11T22:51:08.357 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:51:08.357 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:51:08.357 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:51:08.357 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:51:08.358 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.358 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.358 INFO:tasks.workunit.client.0.smithi059.stdout: "pg": "1.0", 2020-03-11T22:51:08.358 INFO:tasks.workunit.client.0.smithi059.stdout: "currently": "Started/Primary/Active/Clean", 2020-03-11T22:51:08.358 INFO:tasks.workunit.client.0.smithi059.stdout: "history": [ 2020-03-11T22:51:08.358 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.358 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": "27", 2020-03-11T22:51:08.359 INFO:tasks.workunit.client.0.smithi059.stdout: "states": [ 2020-03-11T22:51:08.359 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.359 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Initial", 2020-03-11T22:51:08.359 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:30.668243+0000", 2020-03-11T22:51:08.359 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:47:30.668361+0000" 2020-03-11T22:51:08.359 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:51:08.360 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:51:08.360 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.360 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.360 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": "27", 2020-03-11T22:51:08.360 INFO:tasks.workunit.client.0.smithi059.stdout: "states": [ 2020-03-11T22:51:08.360 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.360 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Reset", 2020-03-11T22:51:08.361 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:30.668362+0000", 2020-03-11T22:51:08.361 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:47:30.668396+0000" 2020-03-11T22:51:08.361 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:51:08.361 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:51:08.361 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.361 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.361 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": "30", 2020-03-11T22:51:08.362 INFO:tasks.workunit.client.0.smithi059.stdout: "states": [ 2020-03-11T22:51:08.362 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.362 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Start", 2020-03-11T22:51:08.362 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:30.668405+0000", 2020-03-11T22:51:08.362 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:47:30.668417+0000" 2020-03-11T22:51:08.362 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.362 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.363 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:51:08.363 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:30.668424+0000", 2020-03-11T22:51:08.363 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:47:30.681840+0000" 2020-03-11T22:51:08.363 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.363 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.363 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:51:08.363 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:30.681840+0000", 2020-03-11T22:51:08.363 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:47:30.681909+0000" 2020-03-11T22:51:08.364 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.364 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.364 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:51:08.364 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:30.681909+0000", 2020-03-11T22:51:08.364 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:47:30.681923+0000" 2020-03-11T22:51:08.364 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.364 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.365 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:51:08.365 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:30.681923+0000", 2020-03-11T22:51:08.365 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:47:30.825234+0000" 2020-03-11T22:51:08.365 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.365 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.365 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:51:08.366 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:30.668420+0000", 2020-03-11T22:51:08.366 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:47:30.825243+0000" 2020-03-11T22:51:08.366 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.366 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.366 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:51:08.366 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:30.825409+0000", 2020-03-11T22:51:08.366 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:47:31.354096+0000" 2020-03-11T22:51:08.367 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.367 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.367 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Active/Recovered", 2020-03-11T22:51:08.367 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:31.354096+0000", 2020-03-11T22:51:08.367 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:47:31.354144+0000" 2020-03-11T22:51:08.367 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.367 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.368 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Active/Clean", 2020-03-11T22:51:08.368 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:31.354144+0000", 2020-03-11T22:51:08.368 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:47:32.058611+0000" 2020-03-11T22:51:08.368 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.368 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.368 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Active", 2020-03-11T22:51:08.369 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:30.825244+0000", 2020-03-11T22:51:08.369 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:47:32.058648+0000" 2020-03-11T22:51:08.369 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.369 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.369 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary", 2020-03-11T22:51:08.369 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:30.668417+0000", 2020-03-11T22:51:08.370 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:47:32.058672+0000" 2020-03-11T22:51:08.370 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.370 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.370 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started", 2020-03-11T22:51:08.370 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:30.668396+0000", 2020-03-11T22:51:08.371 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:47:32.058682+0000" 2020-03-11T22:51:08.371 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:51:08.371 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:51:08.371 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.371 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.371 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": "30", 2020-03-11T22:51:08.372 INFO:tasks.workunit.client.0.smithi059.stdout: "states": [ 2020-03-11T22:51:08.372 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.372 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Reset", 2020-03-11T22:51:08.372 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:32.058682+0000", 2020-03-11T22:51:08.372 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:47:32.059024+0000" 2020-03-11T22:51:08.372 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:51:08.372 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:51:08.373 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.373 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.373 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": "105", 2020-03-11T22:51:08.373 INFO:tasks.workunit.client.0.smithi059.stdout: "states": [ 2020-03-11T22:51:08.374 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.374 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Start", 2020-03-11T22:51:08.374 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:32.059032+0000", 2020-03-11T22:51:08.374 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:47:32.059057+0000" 2020-03-11T22:51:08.374 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.375 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.375 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:51:08.375 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:32.059073+0000", 2020-03-11T22:51:08.375 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:47:32.088514+0000" 2020-03-11T22:51:08.376 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.376 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.376 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:51:08.376 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:32.088515+0000", 2020-03-11T22:51:08.376 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:47:32.088633+0000" 2020-03-11T22:51:08.376 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.377 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.377 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:51:08.377 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:32.088634+0000", 2020-03-11T22:51:08.377 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:47:32.088660+0000" 2020-03-11T22:51:08.378 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.378 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.378 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:51:08.378 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:32.088661+0000", 2020-03-11T22:51:08.378 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:47:32.335293+0000" 2020-03-11T22:51:08.378 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.379 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.379 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:51:08.379 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:32.059066+0000", 2020-03-11T22:51:08.379 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:47:32.335313+0000" 2020-03-11T22:51:08.379 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.379 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.379 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:51:08.380 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:32.335515+0000", 2020-03-11T22:51:08.380 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:47:33.257155+0000" 2020-03-11T22:51:08.380 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.380 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.380 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Active/Recovered", 2020-03-11T22:51:08.380 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:33.257156+0000", 2020-03-11T22:51:08.381 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:47:33.257205+0000" 2020-03-11T22:51:08.381 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.381 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.381 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Active/Clean", 2020-03-11T22:51:08.381 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:33.257205+0000", 2020-03-11T22:51:08.381 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:49:22.681073+0000" 2020-03-11T22:51:08.381 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.382 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.382 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Active", 2020-03-11T22:51:08.382 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:32.335314+0000", 2020-03-11T22:51:08.382 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:49:22.681097+0000" 2020-03-11T22:51:08.382 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.382 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.382 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary", 2020-03-11T22:51:08.383 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:32.059058+0000", 2020-03-11T22:51:08.383 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:49:22.681135+0000" 2020-03-11T22:51:08.383 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.383 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.383 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started", 2020-03-11T22:51:08.383 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:47:32.059025+0000", 2020-03-11T22:51:08.383 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:49:22.681139+0000" 2020-03-11T22:51:08.383 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:51:08.384 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:51:08.384 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.384 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.384 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": "105", 2020-03-11T22:51:08.384 INFO:tasks.workunit.client.0.smithi059.stdout: "states": [ 2020-03-11T22:51:08.384 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.384 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Reset", 2020-03-11T22:51:08.384 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:49:22.681140+0000", 2020-03-11T22:51:08.385 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:49:22.681398+0000" 2020-03-11T22:51:08.385 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:51:08.385 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:51:08.385 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.385 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.385 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": "107", 2020-03-11T22:51:08.386 INFO:tasks.workunit.client.0.smithi059.stdout: "states": [ 2020-03-11T22:51:08.386 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.386 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Start", 2020-03-11T22:51:08.386 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:49:22.681402+0000", 2020-03-11T22:51:08.386 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:49:22.681413+0000" 2020-03-11T22:51:08.386 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.386 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.386 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:51:08.387 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:49:22.681421+0000", 2020-03-11T22:51:08.387 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:49:22.714349+0000" 2020-03-11T22:51:08.387 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.387 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.387 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:51:08.387 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:49:22.714350+0000", 2020-03-11T22:51:08.387 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:49:22.714455+0000" 2020-03-11T22:51:08.388 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.388 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.388 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:51:08.388 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:49:22.681417+0000", 2020-03-11T22:51:08.388 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:49:22.714470+0000" 2020-03-11T22:51:08.388 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.389 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.389 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/WaitActingChange", 2020-03-11T22:51:08.389 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:49:22.714471+0000", 2020-03-11T22:51:08.389 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:49:23.098897+0000" 2020-03-11T22:51:08.389 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.389 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.390 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary", 2020-03-11T22:51:08.390 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:49:22.681414+0000", 2020-03-11T22:51:08.390 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:49:23.098916+0000" 2020-03-11T22:51:08.390 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.390 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.390 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started", 2020-03-11T22:51:08.391 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:49:22.681398+0000", 2020-03-11T22:51:08.391 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:49:23.098922+0000" 2020-03-11T22:51:08.391 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:51:08.391 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:51:08.391 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.392 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.392 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": "107", 2020-03-11T22:51:08.392 INFO:tasks.workunit.client.0.smithi059.stdout: "states": [ 2020-03-11T22:51:08.392 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.392 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Reset", 2020-03-11T22:51:08.392 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:49:23.098923+0000", 2020-03-11T22:51:08.392 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:49:23.099137+0000" 2020-03-11T22:51:08.393 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:51:08.393 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:51:08.393 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.393 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.393 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": "121", 2020-03-11T22:51:08.393 INFO:tasks.workunit.client.0.smithi059.stdout: "states": [ 2020-03-11T22:51:08.393 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.394 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Start", 2020-03-11T22:51:08.394 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:49:23.099141+0000", 2020-03-11T22:51:08.394 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:49:23.099156+0000" 2020-03-11T22:51:08.394 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.394 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.394 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:51:08.394 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:49:23.099167+0000", 2020-03-11T22:51:08.394 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:49:23.207692+0000" 2020-03-11T22:51:08.395 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.395 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.395 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:51:08.395 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:49:23.207692+0000", 2020-03-11T22:51:08.395 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:49:23.207779+0000" 2020-03-11T22:51:08.396 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.396 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.396 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:51:08.396 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:49:23.207779+0000", 2020-03-11T22:51:08.396 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:49:23.207802+0000" 2020-03-11T22:51:08.396 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.397 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.397 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:51:08.397 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:49:23.207802+0000", 2020-03-11T22:51:08.397 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:49:23.387709+0000" 2020-03-11T22:51:08.397 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.397 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.398 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:51:08.398 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:49:23.099161+0000", 2020-03-11T22:51:08.398 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:49:23.387721+0000" 2020-03-11T22:51:08.398 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.398 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.399 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:51:08.399 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:49:23.387993+0000", 2020-03-11T22:51:08.399 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:49:23.727989+0000" 2020-03-11T22:51:08.399 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.399 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.399 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Active/WaitLocalBackfillReserved", 2020-03-11T22:51:08.400 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:49:23.727989+0000", 2020-03-11T22:51:08.400 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:50:43.101173+0000" 2020-03-11T22:51:08.400 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.400 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.400 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Active/WaitRemoteBackfillReserved", 2020-03-11T22:51:08.400 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:50:43.101173+0000", 2020-03-11T22:51:08.401 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:50:43.101685+0000" 2020-03-11T22:51:08.401 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.401 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.401 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Active/Backfilling", 2020-03-11T22:51:08.401 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:50:43.101686+0000", 2020-03-11T22:51:08.401 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:51:05.110768+0000" 2020-03-11T22:51:08.402 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.402 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.402 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Active/Recovered", 2020-03-11T22:51:08.402 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:51:05.110769+0000", 2020-03-11T22:51:08.402 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:51:05.110914+0000" 2020-03-11T22:51:08.402 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.402 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.403 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Active/Clean", 2020-03-11T22:51:08.403 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:51:05.110914+0000", 2020-03-11T22:51:08.403 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:51:05.258185+0000" 2020-03-11T22:51:08.403 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.403 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.403 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary/Active", 2020-03-11T22:51:08.403 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:49:23.387721+0000", 2020-03-11T22:51:08.404 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:51:05.258209+0000" 2020-03-11T22:51:08.404 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.404 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.404 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started/Primary", 2020-03-11T22:51:08.404 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:49:23.099157+0000", 2020-03-11T22:51:08.404 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:51:05.258226+0000" 2020-03-11T22:51:08.404 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.405 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.405 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Started", 2020-03-11T22:51:08.405 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:49:23.099137+0000", 2020-03-11T22:51:08.405 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:51:05.258231+0000" 2020-03-11T22:51:08.405 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:51:08.405 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:51:08.406 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:51:08.406 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.406 INFO:tasks.workunit.client.0.smithi059.stdout: "epoch": "121", 2020-03-11T22:51:08.406 INFO:tasks.workunit.client.0.smithi059.stdout: "states": [ 2020-03-11T22:51:08.406 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:51:08.406 INFO:tasks.workunit.client.0.smithi059.stdout: "state": "Reset", 2020-03-11T22:51:08.406 INFO:tasks.workunit.client.0.smithi059.stdout: "enter": "2020-03-11T22:51:05.258231+0000", 2020-03-11T22:51:08.406 INFO:tasks.workunit.client.0.smithi059.stdout: "exit": "2020-03-11T22:51:05.258512+0000" 2020-03-11T22:51:08.407 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:51:08.407 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:51:08.407 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:51:08.407 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:51:08.407 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:51:08.407 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:51:08.407 INFO:tasks.workunit.client.0.smithi059.stdout:} 2020-03-11T22:51:08.408 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:302: TEST_backfill_priority: '[' 0 '!=' 0 ']' 2020-03-11T22:51:08.409 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:306: TEST_backfill_priority: echo TEST PASSED 2020-03-11T22:51:08.409 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:309: TEST_backfill_priority: delete_pool test1 2020-03-11T22:51:08.409 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test1 2020-03-11T22:51:08.409 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test1 test1 --yes-i-really-really-mean-it 2020-03-11T22:51:08.409 INFO:tasks.workunit.client.0.smithi059.stdout:TEST PASSED 2020-03-11T22:51:08.734 INFO:tasks.workunit.client.0.smithi059.stderr:pool 'test1' does not exist 2020-03-11T22:51:08.743 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:310: TEST_backfill_priority: delete_pool test3 2020-03-11T22:51:08.743 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test3 2020-03-11T22:51:08.743 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test3 test3 --yes-i-really-really-mean-it 2020-03-11T22:51:09.277 INFO:tasks.workunit.client.0.smithi059.stderr:pool 'test3' does not exist 2020-03-11T22:51:09.285 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:311: TEST_backfill_priority: delete_pool test7 2020-03-11T22:51:09.285 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test7 2020-03-11T22:51:09.286 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test7 test7 --yes-i-really-really-mean-it 2020-03-11T22:51:09.803 INFO:tasks.workunit.client.0.smithi059.stderr:pool 'test7' does not exist 2020-03-11T22:51:09.810 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:312: TEST_backfill_priority: kill_daemons td/osd-backfill-prio 2020-03-11T22:51:09.811 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:51:09.811 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:51:09.811 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:51:09.812 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:51:09.812 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:51:09.921 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:51:09.921 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:313: TEST_backfill_priority: return 0 2020-03-11T22:51:09.922 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:40: run: teardown td/osd-backfill-prio 2020-03-11T22:51:09.922 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-prio 2020-03-11T22:51:09.922 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:51:09.922 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-prio KILL 2020-03-11T22:51:09.922 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:51:09.922 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:51:09.923 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:51:09.923 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:51:09.923 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:51:09.927 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:51:09.928 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:51:09.928 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:51:09.928 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:51:09.929 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:51:09.929 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:51:09.930 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:51:09.935 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:51:09.935 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:51:09.935 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:51:09.935 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:51:09.935 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:51:09.936 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:51:09.936 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-prio 2020-03-11T22:51:09.959 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:51:09.959 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:51:09.959 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56462 2020-03-11T22:51:09.960 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.56462 2020-03-11T22:51:09.962 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:51:09.962 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:51:09.962 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: code=0 2020-03-11T22:51:09.962 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: teardown td/osd-backfill-prio 0 2020-03-11T22:51:09.963 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-prio 2020-03-11T22:51:09.963 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2020-03-11T22:51:09.963 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-prio KILL 2020-03-11T22:51:09.963 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:51:09.963 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:51:09.963 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:51:09.963 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:51:09.964 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:51:09.964 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:51:09.965 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:51:09.965 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:51:09.965 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:51:09.965 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:51:09.966 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:51:09.966 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:51:09.967 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:51:09.967 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:51:09.967 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:51:09.967 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:51:09.968 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:51:09.969 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2020-03-11T22:51:09.969 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-prio 2020-03-11T22:51:09.970 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:51:09.970 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:51:09.970 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56462 2020-03-11T22:51:09.970 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.56462 2020-03-11T22:51:09.971 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:51:09.971 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:51:09.972 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: return 0 2020-03-11T22:51:09.972 INFO:teuthology.orchestra.run:Running command with timeout 3600 2020-03-11T22:51:09.972 INFO:teuthology.orchestra.run.smithi059:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2020-03-11T22:51:10.039 INFO:tasks.workunit:Running workunit osd/osd-backfill-recovery-log.sh... 2020-03-11T22:51:10.040 INFO:teuthology.orchestra.run.smithi059:workunit test osd/osd-backfill-recovery-log.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4acc1121e47039a3cdf2978ae458c10d153aa141 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh 2020-03-11T22:51:10.071 INFO:tasks.workunit.client.0.smithi059.stderr:stty: 'standard input': Inappropriate ioctl for device 2020-03-11T22:51:10.078 INFO:tasks.workunit.client.0.smithi059.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2020-03-11T22:51:10.079 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2020-03-11T22:51:10.079 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2020-03-11T22:51:10.079 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: main: export PYTHONWARNINGS=ignore 2020-03-11T22:51:10.079 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: main: PYTHONWARNINGS=ignore 2020-03-11T22:51:10.080 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: main: export CEPH_CONF=/dev/null 2020-03-11T22:51:10.080 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: main: CEPH_CONF=/dev/null 2020-03-11T22:51:10.080 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: main: unset CEPH_ARGS 2020-03-11T22:51:10.080 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: local code 2020-03-11T22:51:10.080 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: run td/osd-backfill-recovery-log 2020-03-11T22:51:10.080 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:21: run: local dir=td/osd-backfill-recovery-log 2020-03-11T22:51:10.081 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:22: run: shift 2020-03-11T22:51:10.081 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:25: run: export CEPH_MON=127.0.0.1:7129 2020-03-11T22:51:10.081 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:25: run: CEPH_MON=127.0.0.1:7129 2020-03-11T22:51:10.081 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:26: run: export CEPH_ARGS 2020-03-11T22:51:10.081 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:27: run: uuidgen 2020-03-11T22:51:10.081 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:27: run: CEPH_ARGS+='--fsid=7f0bc282-eebc-452f-ae5f-6537699866e7 --auth-supported=none ' 2020-03-11T22:51:10.081 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:51:10.082 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:30: run: set 2020-03-11T22:51:10.082 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2020-03-11T22:51:10.084 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:30: run: local 'funcs=TEST_backfill_log_1 2020-03-11T22:51:10.084 INFO:tasks.workunit.client.0.smithi059.stderr:TEST_backfill_log_2 2020-03-11T22:51:10.084 INFO:tasks.workunit.client.0.smithi059.stderr:TEST_recovery_1 2020-03-11T22:51:10.084 INFO:tasks.workunit.client.0.smithi059.stderr:TEST_recovery_2' 2020-03-11T22:51:10.084 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:31: run: for func in $funcs 2020-03-11T22:51:10.085 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:32: run: setup td/osd-backfill-recovery-log 2020-03-11T22:51:10.085 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-recovery-log 2020-03-11T22:51:10.085 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-recovery-log 2020-03-11T22:51:10.085 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-recovery-log 2020-03-11T22:51:10.085 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:51:10.085 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-recovery-log KILL 2020-03-11T22:51:10.085 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:51:10.086 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:51:10.086 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:51:10.086 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:51:10.086 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:51:10.086 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:51:10.086 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:51:10.087 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:51:10.087 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:51:10.087 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:51:10.087 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:51:10.087 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:51:10.088 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:51:10.089 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:51:10.089 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:51:10.089 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:51:10.089 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:51:10.090 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:51:10.090 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-recovery-log 2020-03-11T22:51:10.090 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:51:10.090 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:51:10.090 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76608 2020-03-11T22:51:10.090 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76608 2020-03-11T22:51:10.093 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:51:10.094 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:51:10.094 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-recovery-log 2020-03-11T22:51:10.094 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-11T22:51:10.094 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:51:10.094 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76608 2020-03-11T22:51:10.094 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76608 2020-03-11T22:51:10.094 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-11T22:51:10.095 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2020-03-11T22:51:10.095 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2020-03-11T22:51:10.095 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-11T22:51:10.095 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-recovery-log 1' TERM HUP INT 2020-03-11T22:51:10.095 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:33: run: TEST_backfill_log_1 td/osd-backfill-recovery-log 2020-03-11T22:51:10.095 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:103: TEST_backfill_log_1: local dir=td/osd-backfill-recovery-log 2020-03-11T22:51:10.095 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:105: TEST_backfill_log_1: _common_test td/osd-backfill-recovery-log '--osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2 8 150 2020-03-11T22:51:10.096 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:40: _common_test: local dir=td/osd-backfill-recovery-log 2020-03-11T22:51:10.096 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:41: _common_test: local 'extra_opts=--osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-03-11T22:51:10.096 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:42: _common_test: local loglen=2 2020-03-11T22:51:10.096 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:43: _common_test: local dupslen=8 2020-03-11T22:51:10.096 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:44: _common_test: local objects=150 2020-03-11T22:51:10.096 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:45: _common_test: local moreobjects=0 2020-03-11T22:51:10.097 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:47: _common_test: local OSDS=6 2020-03-11T22:51:10.097 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:49: _common_test: run_mon td/osd-backfill-recovery-log a 2020-03-11T22:51:10.097 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-recovery-log 2020-03-11T22:51:10.097 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-11T22:51:10.097 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-11T22:51:10.097 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-11T22:51:10.098 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-recovery-log/a 2020-03-11T22:51:10.098 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-recovery-log/a --run-dir=td/osd-backfill-recovery-log 2020-03-11T22:51:10.413 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-11T22:51:10.413 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:51:10.413 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:51:10.413 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:51:10.414 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:51:10.414 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76608 2020-03-11T22:51:10.414 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76608/$cluster-$name.asok' 2020-03-11T22:51:10.414 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-recovery-log/a '--log-file=td/osd-backfill-recovery-log/$name.log' '--admin-socket=/tmp/ceph-asok.76608/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-recovery-log/log --run-dir=td/osd-backfill-recovery-log '--pid-file=td/osd-backfill-recovery-log/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-11T22:51:10.580 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-11T22:51:10.582 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-11T22:51:10.582 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:51:10.582 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:51:10.582 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-11T22:51:10.582 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-11T22:51:10.583 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:51:10.583 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:51:10.583 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:51:10.583 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:51:10.583 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:51:10.583 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76608 2020-03-11T22:51:10.583 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76608/ceph-mon.a.asok 2020-03-11T22:51:10.584 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:51:10.584 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.76608/ceph-mon.a.asok config get fsid 2020-03-11T22:51:10.675 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-11T22:51:10.675 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:51:10.675 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:51:10.676 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-11T22:51:10.676 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-11T22:51:10.676 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:51:10.676 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:51:10.676 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:51:10.676 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:51:10.676 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:51:10.677 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76608 2020-03-11T22:51:10.677 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76608/ceph-mon.a.asok 2020-03-11T22:51:10.677 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:51:10.677 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.76608/ceph-mon.a.asok config get mon_host 2020-03-11T22:51:10.765 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:50: _common_test: run_mgr td/osd-backfill-recovery-log x 2020-03-11T22:51:10.765 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/osd-backfill-recovery-log 2020-03-11T22:51:10.765 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-11T22:51:10.766 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-11T22:51:10.766 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-11T22:51:10.766 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/osd-backfill-recovery-log/x 2020-03-11T22:51:10.766 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-11T22:51:11.683 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-11T22:51:11.683 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:51:11.683 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:51:11.684 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:51:11.684 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:51:11.684 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76608 2020-03-11T22:51:11.684 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76608/$cluster-$name.asok' 2020-03-11T22:51:11.684 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:51:11.685 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-recovery-log/x '--log-file=td/osd-backfill-recovery-log/$name.log' '--admin-socket=/tmp/ceph-asok.76608/$cluster-$name.asok' --run-dir=td/osd-backfill-recovery-log '--pid-file=td/osd-backfill-recovery-log/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:51:11.706 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:51: _common_test: export CEPH_ARGS 2020-03-11T22:51:11.706 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:52: _common_test: export 'EXTRA_OPTS= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-03-11T22:51:11.706 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:52: _common_test: EXTRA_OPTS=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-03-11T22:51:11.706 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: expr 6 - 1 2020-03-11T22:51:11.707 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: seq 0 5 2020-03-11T22:51:11.711 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:51:11.711 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 0 2020-03-11T22:51:11.712 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:51:11.712 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:51:11.712 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-11T22:51:11.712 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:51:11.712 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/0 2020-03-11T22:51:11.712 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=7f0bc282-eebc-452f-ae5f-6537699866e7 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:51:11.713 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:51:11.713 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:51:11.713 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:51:11.713 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/0' 2020-03-11T22:51:11.713 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/0/journal' 2020-03-11T22:51:11.713 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:51:11.713 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-03-11T22:51:11.714 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:51:11.714 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:51:11.714 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:51:11.714 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:51:11.714 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:51:11.714 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:51:11.714 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76608 2020-03-11T22:51:11.714 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76608/$cluster-$name.asok' 2020-03-11T22:51:11.715 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76608/$cluster-$name.asok' 2020-03-11T22:51:11.715 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:51:11.715 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:51:11.715 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:51:11.715 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:51:11.716 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:51:11.716 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:51:11.716 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:51:11.716 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:51:11.716 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:51:11.716 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:51:11.716 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/0 2020-03-11T22:51:11.717 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:51:11.717 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=5719524c-4414-45a6-9221-ca3792a34c12 2020-03-11T22:51:11.717 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 5719524c-4414-45a6-9221-ca3792a34c12' 2020-03-11T22:51:11.717 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:51:11.717 INFO:tasks.workunit.client.0.smithi059.stdout:add osd0 5719524c-4414-45a6-9221-ca3792a34c12 2020-03-11T22:51:11.725 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBfa2le4GQTKxAAf/qo5Hn6FdKCW81yh3VUSg== 2020-03-11T22:51:11.726 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBfa2le4GQTKxAAf/qo5Hn6FdKCW81yh3VUSg=="}' 2020-03-11T22:51:11.726 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 5719524c-4414-45a6-9221-ca3792a34c12 -i td/osd-backfill-recovery-log/0/new.json 2020-03-11T22:51:12.055 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:51:12.065 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/0/new.json 2020-03-11T22:51:12.066 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=7f0bc282-eebc-452f-ae5f-6537699866e7 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/0 --osd-journal=td/osd-backfill-recovery-log/0/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76608/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBfa2le4GQTKxAAf/qo5Hn6FdKCW81yh3VUSg== --osd-uuid 5719524c-4414-45a6-9221-ca3792a34c12 2020-03-11T22:51:12.091 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:51:12.087+0000 7f13633ebec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:12.092 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:51:12.089+0000 7f13633ebec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:12.094 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:51:12.091+0000 7f13633ebec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:12.095 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:51:12.091+0000 7f13633ebec0 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2020-03-11T22:51:12.095 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:51:12.091+0000 7f13633ebec0 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2020-03-11T22:51:12.095 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:51:12.091+0000 7f13633ebec0 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2020-03-11T22:51:12.187 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:51:12.184+0000 7f13633ebec0 -1 bluestore(td/osd-backfill-recovery-log/0) _read_fsid unparsable uuid 2020-03-11T22:51:15.298 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/0/keyring 2020-03-11T22:51:15.298 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:51:15.299 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-11T22:51:15.299 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:51:15.299 INFO:tasks.workunit.client.0.smithi059.stdout:adding osd0 key to auth repository 2020-03-11T22:51:15.780 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-11T22:51:15.781 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=7f0bc282-eebc-452f-ae5f-6537699866e7 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/0 --osd-journal=td/osd-backfill-recovery-log/0/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76608/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:51:15.781 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:51:15.781 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:51:15.781 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:51:15.781 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.0 2020-03-11T22:51:15.803 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:51:15.799+0000 7fa68a263ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:15.805 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:51:15.801+0000 7fa68a263ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:15.807 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:51:15.804+0000 7fa68a263ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:16.150 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-11T22:51:16.150 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:51:16.150 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:51:16.150 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:51:16.150 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:51:16.151 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:51:16.151 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:51:16.151 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:51:16.151 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:51:16.151 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:51:16.326 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:51:16.322+0000 7fa68a263ec0 -1 Falling back to public interface 2020-03-11T22:51:16.516 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:51:17.095 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:51:17.091+0000 7fa68a263ec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-11T22:51:17.518 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:51:17.518 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:51:17.518 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:51:17.518 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:51:17.519 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:51:17.519 INFO:tasks.workunit.client.0.smithi059.stdout:1 2020-03-11T22:51:17.902 INFO:tasks.workunit.client.0.smithi059.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/77040,v1:127.0.0.1:6803/77040] [v2:127.0.0.1:6804/77040,v1:127.0.0.1:6805/77040] exists,up 5719524c-4414-45a6-9221-ca3792a34c12 2020-03-11T22:51:17.902 INFO:tasks.workunit.client.0.smithi059.stdout:add osd1 5ac030b0-87b8-40bb-9f4e-64c0809aeaf6 2020-03-11T22:51:17.903 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:51:17.903 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:51:17.903 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:51:17.903 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:51:17.903 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 1 2020-03-11T22:51:17.903 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:51:17.904 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:51:17.904 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-11T22:51:17.904 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:51:17.904 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/1 2020-03-11T22:51:17.904 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=7f0bc282-eebc-452f-ae5f-6537699866e7 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:51:17.904 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:51:17.904 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:51:17.904 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:51:17.905 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/1' 2020-03-11T22:51:17.905 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/1/journal' 2020-03-11T22:51:17.905 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:51:17.905 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-03-11T22:51:17.905 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:51:17.905 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:51:17.905 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:51:17.906 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:51:17.906 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:51:17.906 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:51:17.906 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76608 2020-03-11T22:51:17.906 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76608/$cluster-$name.asok' 2020-03-11T22:51:17.906 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76608/$cluster-$name.asok' 2020-03-11T22:51:17.907 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:51:17.907 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:51:17.907 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:51:17.907 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:51:17.907 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:51:17.907 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:51:17.907 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:51:17.907 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:51:17.908 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:51:17.908 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:51:17.908 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/1 2020-03-11T22:51:17.908 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:51:17.908 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=5ac030b0-87b8-40bb-9f4e-64c0809aeaf6 2020-03-11T22:51:17.908 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 5ac030b0-87b8-40bb-9f4e-64c0809aeaf6' 2020-03-11T22:51:17.908 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:51:17.916 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBla2le4ed0NhAAobwsWG2yfVTP5freYCP9yg== 2020-03-11T22:51:17.917 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBla2le4ed0NhAAobwsWG2yfVTP5freYCP9yg=="}' 2020-03-11T22:51:17.917 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 5ac030b0-87b8-40bb-9f4e-64c0809aeaf6 -i td/osd-backfill-recovery-log/1/new.json 2020-03-11T22:51:18.489 INFO:tasks.workunit.client.0.smithi059.stdout:1 2020-03-11T22:51:18.498 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/1/new.json 2020-03-11T22:51:18.499 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=7f0bc282-eebc-452f-ae5f-6537699866e7 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/1 --osd-journal=td/osd-backfill-recovery-log/1/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76608/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBla2le4ed0NhAAobwsWG2yfVTP5freYCP9yg== --osd-uuid 5ac030b0-87b8-40bb-9f4e-64c0809aeaf6 2020-03-11T22:51:18.525 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:51:18.521+0000 7f389b070ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:18.560 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:51:18.556+0000 7f389b070ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:18.561 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:51:18.557+0000 7f389b070ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:18.561 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:51:18.558+0000 7f389b070ec0 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2020-03-11T22:51:18.562 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:51:18.558+0000 7f389b070ec0 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2020-03-11T22:51:18.562 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:51:18.558+0000 7f389b070ec0 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2020-03-11T22:51:18.660 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:51:18.656+0000 7f389b070ec0 -1 bluestore(td/osd-backfill-recovery-log/1) _read_fsid unparsable uuid 2020-03-11T22:51:22.018 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/1/keyring 2020-03-11T22:51:22.018 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:51:22.019 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-03-11T22:51:22.019 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:51:22.020 INFO:tasks.workunit.client.0.smithi059.stdout:adding osd1 key to auth repository 2020-03-11T22:51:22.502 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.1 2020-03-11T22:51:22.502 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-03-11T22:51:22.502 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=7f0bc282-eebc-452f-ae5f-6537699866e7 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/1 --osd-journal=td/osd-backfill-recovery-log/1/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76608/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:51:22.502 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:51:22.502 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:51:22.503 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:51:22.525 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:51:22.521+0000 7fa10c2d7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:22.526 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:51:22.523+0000 7fa10c2d7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:22.527 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:51:22.524+0000 7fa10c2d7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:22.886 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-03-11T22:51:22.886 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:51:22.886 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:51:22.886 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:51:22.886 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:51:22.886 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:51:22.887 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:51:22.887 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:51:22.887 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:51:22.887 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:51:23.043 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:51:23.039+0000 7fa10c2d7ec0 -1 Falling back to public interface 2020-03-11T22:51:23.336 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:51:23.792 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:51:23.788+0000 7fa10c2d7ec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-11T22:51:24.338 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:51:24.338 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:51:24.338 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:51:24.338 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:51:24.339 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:51:24.339 INFO:tasks.workunit.client.0.smithi059.stdout:1 2020-03-11T22:51:24.749 INFO:tasks.workunit.client.0.smithi059.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/77395,v1:127.0.0.1:6811/77395] [v2:127.0.0.1:6812/77395,v1:127.0.0.1:6813/77395] exists,up 5ac030b0-87b8-40bb-9f4e-64c0809aeaf6 2020-03-11T22:51:24.750 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:51:24.750 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:51:24.750 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:51:24.750 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:51:24.751 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 2 2020-03-11T22:51:24.751 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:51:24.751 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:51:24.751 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-03-11T22:51:24.751 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:51:24.751 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/2 2020-03-11T22:51:24.751 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=7f0bc282-eebc-452f-ae5f-6537699866e7 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:51:24.752 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:51:24.752 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:51:24.752 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:51:24.752 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/2' 2020-03-11T22:51:24.752 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/2/journal' 2020-03-11T22:51:24.752 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:51:24.752 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-03-11T22:51:24.752 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:51:24.753 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:51:24.753 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:51:24.753 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:51:24.753 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:51:24.753 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:51:24.753 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76608 2020-03-11T22:51:24.753 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76608/$cluster-$name.asok' 2020-03-11T22:51:24.754 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76608/$cluster-$name.asok' 2020-03-11T22:51:24.754 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:51:24.754 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:51:24.754 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:51:24.754 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:51:24.754 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:51:24.754 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:51:24.755 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:51:24.755 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:51:24.755 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:51:24.755 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:51:24.755 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/2 2020-03-11T22:51:24.756 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:51:24.756 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=1c827dba-e823-4623-9df2-db488afaa445 2020-03-11T22:51:24.756 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 1c827dba-e823-4623-9df2-db488afaa445' 2020-03-11T22:51:24.756 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:51:24.756 INFO:tasks.workunit.client.0.smithi059.stdout:add osd2 1c827dba-e823-4623-9df2-db488afaa445 2020-03-11T22:51:24.766 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBsa2leTdx9LRAAFwpz/7biNXYYW2RuLdiZXA== 2020-03-11T22:51:24.766 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBsa2leTdx9LRAAFwpz/7biNXYYW2RuLdiZXA=="}' 2020-03-11T22:51:24.766 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 1c827dba-e823-4623-9df2-db488afaa445 -i td/osd-backfill-recovery-log/2/new.json 2020-03-11T22:51:25.248 INFO:tasks.workunit.client.0.smithi059.stdout:2 2020-03-11T22:51:25.258 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/2/new.json 2020-03-11T22:51:25.259 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=7f0bc282-eebc-452f-ae5f-6537699866e7 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/2 --osd-journal=td/osd-backfill-recovery-log/2/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76608/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBsa2leTdx9LRAAFwpz/7biNXYYW2RuLdiZXA== --osd-uuid 1c827dba-e823-4623-9df2-db488afaa445 2020-03-11T22:51:25.283 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:51:25.281+0000 7fbbbd709ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:25.315 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:51:25.313+0000 7fbbbd709ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:25.318 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:51:25.315+0000 7fbbbd709ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:25.318 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:51:25.315+0000 7fbbbd709ec0 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2020-03-11T22:51:25.318 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:51:25.315+0000 7fbbbd709ec0 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2020-03-11T22:51:25.318 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:51:25.315+0000 7fbbbd709ec0 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2020-03-11T22:51:25.374 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:51:25.371+0000 7fbbbd709ec0 -1 bluestore(td/osd-backfill-recovery-log/2) _read_fsid unparsable uuid 2020-03-11T22:51:28.517 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/2/keyring 2020-03-11T22:51:28.517 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:51:28.518 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-03-11T22:51:28.518 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:51:28.519 INFO:tasks.workunit.client.0.smithi059.stdout:adding osd2 key to auth repository 2020-03-11T22:51:29.005 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-03-11T22:51:29.005 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=7f0bc282-eebc-452f-ae5f-6537699866e7 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/2 --osd-journal=td/osd-backfill-recovery-log/2/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76608/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:51:29.005 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:51:29.005 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:51:29.005 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:51:29.006 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.2 2020-03-11T22:51:29.027 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:51:29.024+0000 7f1bf8509ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:29.029 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:51:29.026+0000 7f1bf8509ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:29.031 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:51:29.028+0000 7f1bf8509ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:29.370 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-03-11T22:51:29.370 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:51:29.370 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:51:29.370 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:51:29.370 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:51:29.371 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:51:29.371 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:51:29.371 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:51:29.371 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:51:29.371 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:51:29.546 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:51:29.543+0000 7f1bf8509ec0 -1 Falling back to public interface 2020-03-11T22:51:29.735 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:51:30.281 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:51:30.278+0000 7f1bf8509ec0 -1 osd.2 0 log_to_monitors {default=true} 2020-03-11T22:51:30.737 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:51:30.737 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:51:30.737 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:51:30.737 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:51:30.737 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:51:30.738 INFO:tasks.workunit.client.0.smithi059.stdout:1 2020-03-11T22:51:31.137 INFO:tasks.workunit.client.0.smithi059.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/77750,v1:127.0.0.1:6819/77750] [v2:127.0.0.1:6820/77750,v1:127.0.0.1:6821/77750] exists,up 1c827dba-e823-4623-9df2-db488afaa445 2020-03-11T22:51:31.137 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:51:31.138 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:51:31.138 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:51:31.138 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:51:31.138 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 3 2020-03-11T22:51:31.138 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:51:31.138 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:51:31.138 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=3 2020-03-11T22:51:31.138 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:51:31.139 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/3 2020-03-11T22:51:31.139 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=7f0bc282-eebc-452f-ae5f-6537699866e7 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:51:31.139 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:51:31.139 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:51:31.139 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:51:31.139 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/3' 2020-03-11T22:51:31.139 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/3/journal' 2020-03-11T22:51:31.139 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:51:31.140 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-03-11T22:51:31.140 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:51:31.140 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:51:31.140 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:51:31.140 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:51:31.140 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:51:31.140 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:51:31.140 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76608 2020-03-11T22:51:31.141 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76608/$cluster-$name.asok' 2020-03-11T22:51:31.141 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76608/$cluster-$name.asok' 2020-03-11T22:51:31.141 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:51:31.141 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:51:31.141 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:51:31.142 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:51:31.142 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:51:31.142 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:51:31.142 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:51:31.142 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:51:31.143 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:51:31.143 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:51:31.143 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/3 2020-03-11T22:51:31.144 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:51:31.144 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=fa6d5779-c480-47a4-9525-2356fd8327f8 2020-03-11T22:51:31.144 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd3 fa6d5779-c480-47a4-9525-2356fd8327f8' 2020-03-11T22:51:31.144 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:51:31.144 INFO:tasks.workunit.client.0.smithi059.stdout:add osd3 fa6d5779-c480-47a4-9525-2356fd8327f8 2020-03-11T22:51:31.153 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBza2leS9QACRAApya0zpCU9Nl6tsSjKgPhNg== 2020-03-11T22:51:31.153 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBza2leS9QACRAApya0zpCU9Nl6tsSjKgPhNg=="}' 2020-03-11T22:51:31.154 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new fa6d5779-c480-47a4-9525-2356fd8327f8 -i td/osd-backfill-recovery-log/3/new.json 2020-03-11T22:51:31.679 INFO:tasks.workunit.client.0.smithi059.stdout:3 2020-03-11T22:51:31.689 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/3/new.json 2020-03-11T22:51:31.690 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 3 --fsid=7f0bc282-eebc-452f-ae5f-6537699866e7 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/3 --osd-journal=td/osd-backfill-recovery-log/3/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76608/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBza2leS9QACRAApya0zpCU9Nl6tsSjKgPhNg== --osd-uuid fa6d5779-c480-47a4-9525-2356fd8327f8 2020-03-11T22:51:31.714 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:51:31.711+0000 7f48cc863ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:31.716 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:51:31.713+0000 7f48cc863ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:31.718 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:51:31.715+0000 7f48cc863ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:31.718 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:51:31.715+0000 7f48cc863ec0 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2020-03-11T22:51:31.718 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:51:31.715+0000 7f48cc863ec0 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2020-03-11T22:51:31.719 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:51:31.715+0000 7f48cc863ec0 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2020-03-11T22:51:31.801 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:51:31.798+0000 7f48cc863ec0 -1 bluestore(td/osd-backfill-recovery-log/3) _read_fsid unparsable uuid 2020-03-11T22:51:34.939 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/3/keyring 2020-03-11T22:51:34.939 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:51:34.940 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd3 key to auth repository 2020-03-11T22:51:34.940 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:51:34.940 INFO:tasks.workunit.client.0.smithi059.stdout:adding osd3 key to auth repository 2020-03-11T22:51:35.440 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.3 2020-03-11T22:51:35.440 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 3 --fsid=7f0bc282-eebc-452f-ae5f-6537699866e7 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/3 --osd-journal=td/osd-backfill-recovery-log/3/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76608/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:51:35.441 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:51:35.441 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:51:35.441 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:51:35.441 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.3 2020-03-11T22:51:35.463 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:51:35.460+0000 7f06b093aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:35.464 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:51:35.462+0000 7f06b093aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:35.466 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:51:35.464+0000 7f06b093aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:35.838 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 3 2020-03-11T22:51:35.838 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:51:35.838 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=3 2020-03-11T22:51:35.838 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:51:35.839 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:51:35.839 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:51:35.839 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:51:35.839 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:51:35.839 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:51:35.839 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:51:35.990 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:51:35.987+0000 7f06b093aec0 -1 Falling back to public interface 2020-03-11T22:51:36.199 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:51:36.772 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:51:36.769+0000 7f06b093aec0 -1 osd.3 0 log_to_monitors {default=true} 2020-03-11T22:51:37.201 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:51:37.201 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:51:37.201 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:51:37.201 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:51:37.201 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:51:37.202 INFO:tasks.workunit.client.0.smithi059.stdout:1 2020-03-11T22:51:37.611 INFO:tasks.workunit.client.0.smithi059.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/78105,v1:127.0.0.1:6827/78105] [v2:127.0.0.1:6828/78105,v1:127.0.0.1:6829/78105] exists,up fa6d5779-c480-47a4-9525-2356fd8327f8 2020-03-11T22:51:37.611 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:51:37.611 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:51:37.612 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:51:37.612 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:51:37.612 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 4 2020-03-11T22:51:37.612 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:51:37.612 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:51:37.612 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=4 2020-03-11T22:51:37.613 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:51:37.613 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/4 2020-03-11T22:51:37.613 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=7f0bc282-eebc-452f-ae5f-6537699866e7 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:51:37.613 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:51:37.613 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:51:37.613 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:51:37.613 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/4' 2020-03-11T22:51:37.614 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/4/journal' 2020-03-11T22:51:37.614 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:51:37.614 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-03-11T22:51:37.614 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:51:37.614 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:51:37.614 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:51:37.614 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:51:37.615 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:51:37.615 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:51:37.615 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76608 2020-03-11T22:51:37.615 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76608/$cluster-$name.asok' 2020-03-11T22:51:37.615 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76608/$cluster-$name.asok' 2020-03-11T22:51:37.615 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:51:37.615 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:51:37.616 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:51:37.616 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:51:37.616 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:51:37.616 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:51:37.616 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:51:37.616 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:51:37.617 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:51:37.617 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:51:37.617 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/4 2020-03-11T22:51:37.618 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:51:37.618 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=02ed8266-4ce1-4efd-bfe6-6fe391920821 2020-03-11T22:51:37.618 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd4 02ed8266-4ce1-4efd-bfe6-6fe391920821' 2020-03-11T22:51:37.618 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:51:37.618 INFO:tasks.workunit.client.0.smithi059.stdout:add osd4 02ed8266-4ce1-4efd-bfe6-6fe391920821 2020-03-11T22:51:37.627 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQB5a2lei4o9JRAAmCZaB/YXRI6xeqFskuY/Fw== 2020-03-11T22:51:37.627 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQB5a2lei4o9JRAAmCZaB/YXRI6xeqFskuY/Fw=="}' 2020-03-11T22:51:37.627 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 02ed8266-4ce1-4efd-bfe6-6fe391920821 -i td/osd-backfill-recovery-log/4/new.json 2020-03-11T22:51:38.192 INFO:tasks.workunit.client.0.smithi059.stdout:4 2020-03-11T22:51:38.202 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/4/new.json 2020-03-11T22:51:38.203 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 4 --fsid=7f0bc282-eebc-452f-ae5f-6537699866e7 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/4 --osd-journal=td/osd-backfill-recovery-log/4/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76608/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQB5a2lei4o9JRAAmCZaB/YXRI6xeqFskuY/Fw== --osd-uuid 02ed8266-4ce1-4efd-bfe6-6fe391920821 2020-03-11T22:51:38.228 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:51:38.225+0000 7f265be4eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:38.263 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:51:38.260+0000 7f265be4eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:38.265 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:51:38.261+0000 7f265be4eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:38.265 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:51:38.262+0000 7f265be4eec0 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2020-03-11T22:51:38.265 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:51:38.262+0000 7f265be4eec0 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2020-03-11T22:51:38.265 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:51:38.262+0000 7f265be4eec0 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2020-03-11T22:51:38.371 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:51:38.368+0000 7f265be4eec0 -1 bluestore(td/osd-backfill-recovery-log/4) _read_fsid unparsable uuid 2020-03-11T22:51:41.548 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/4/keyring 2020-03-11T22:51:41.548 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:51:41.549 INFO:tasks.workunit.client.0.smithi059.stdout:adding osd4 key to auth repository 2020-03-11T22:51:41.549 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd4 key to auth repository 2020-03-11T22:51:41.550 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:51:42.039 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.4 2020-03-11T22:51:42.040 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:51:42.040 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 4 --fsid=7f0bc282-eebc-452f-ae5f-6537699866e7 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/4 --osd-journal=td/osd-backfill-recovery-log/4/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76608/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:51:42.040 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:51:42.040 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:51:42.040 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.4 2020-03-11T22:51:42.062 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:51:42.059+0000 7f5248b87ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:42.097 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:51:42.094+0000 7f5248b87ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:42.098 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:51:42.095+0000 7f5248b87ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:42.406 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 4 2020-03-11T22:51:42.406 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:51:42.406 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=4 2020-03-11T22:51:42.406 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:51:42.407 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:51:42.407 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:51:42.407 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:51:42.407 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:51:42.407 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.4 up' 2020-03-11T22:51:42.407 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:51:42.616 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:51:42.612+0000 7f5248b87ec0 -1 Falling back to public interface 2020-03-11T22:51:42.775 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:51:43.464 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:51:43.461+0000 7f5248b87ec0 -1 osd.4 0 log_to_monitors {default=true} 2020-03-11T22:51:43.777 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:51:43.777 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:51:43.777 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:51:43.777 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:51:43.777 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.4 up' 2020-03-11T22:51:43.778 INFO:tasks.workunit.client.0.smithi059.stdout:1 2020-03-11T22:51:44.179 INFO:tasks.workunit.client.0.smithi059.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/78468,v1:127.0.0.1:6835/78468] [v2:127.0.0.1:6836/78468,v1:127.0.0.1:6837/78468] exists,up 02ed8266-4ce1-4efd-bfe6-6fe391920821 2020-03-11T22:51:44.179 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:51:44.179 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:51:44.179 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:51:44.179 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:51:44.180 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 5 2020-03-11T22:51:44.180 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:51:44.180 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:51:44.180 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=5 2020-03-11T22:51:44.180 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:51:44.180 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/5 2020-03-11T22:51:44.180 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=7f0bc282-eebc-452f-ae5f-6537699866e7 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:51:44.180 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:51:44.181 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:51:44.181 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:51:44.181 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/5' 2020-03-11T22:51:44.181 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/5/journal' 2020-03-11T22:51:44.181 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:51:44.181 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-03-11T22:51:44.182 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:51:44.182 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:51:44.182 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:51:44.182 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:51:44.182 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:51:44.182 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:51:44.182 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76608 2020-03-11T22:51:44.183 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76608/$cluster-$name.asok' 2020-03-11T22:51:44.183 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76608/$cluster-$name.asok' 2020-03-11T22:51:44.183 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:51:44.183 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:51:44.183 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:51:44.183 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:51:44.184 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:51:44.184 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:51:44.184 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:51:44.184 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:51:44.184 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:51:44.184 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:51:44.185 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/5 2020-03-11T22:51:44.185 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:51:44.186 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=4acde14e-f4ad-4c62-9aa9-bb310069c29b 2020-03-11T22:51:44.186 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd5 4acde14e-f4ad-4c62-9aa9-bb310069c29b' 2020-03-11T22:51:44.186 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:51:44.187 INFO:tasks.workunit.client.0.smithi059.stdout:add osd5 4acde14e-f4ad-4c62-9aa9-bb310069c29b 2020-03-11T22:51:44.194 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQCAa2leVsRqCxAAYQK96pjOTitXcqBGVmrywg== 2020-03-11T22:51:44.194 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQCAa2leVsRqCxAAYQK96pjOTitXcqBGVmrywg=="}' 2020-03-11T22:51:44.194 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 4acde14e-f4ad-4c62-9aa9-bb310069c29b -i td/osd-backfill-recovery-log/5/new.json 2020-03-11T22:51:44.719 INFO:tasks.workunit.client.0.smithi059.stdout:5 2020-03-11T22:51:44.729 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/5/new.json 2020-03-11T22:51:44.730 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 5 --fsid=7f0bc282-eebc-452f-ae5f-6537699866e7 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/5 --osd-journal=td/osd-backfill-recovery-log/5/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76608/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCAa2leVsRqCxAAYQK96pjOTitXcqBGVmrywg== --osd-uuid 4acde14e-f4ad-4c62-9aa9-bb310069c29b 2020-03-11T22:51:44.755 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:51:44.752+0000 7fc999260ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:44.757 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:51:44.754+0000 7fc999260ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:44.759 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:51:44.756+0000 7fc999260ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:44.760 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:51:44.756+0000 7fc999260ec0 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2020-03-11T22:51:44.760 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:51:44.756+0000 7fc999260ec0 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2020-03-11T22:51:44.760 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:51:44.756+0000 7fc999260ec0 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2020-03-11T22:51:44.795 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:51:44.792+0000 7fc999260ec0 -1 bluestore(td/osd-backfill-recovery-log/5) _read_fsid unparsable uuid 2020-03-11T22:51:48.124 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/5/keyring 2020-03-11T22:51:48.124 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:51:48.125 INFO:tasks.workunit.client.0.smithi059.stdout:adding osd5 key to auth repository 2020-03-11T22:51:48.125 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd5 key to auth repository 2020-03-11T22:51:48.125 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:51:48.614 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.5 2020-03-11T22:51:48.615 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.5 2020-03-11T22:51:48.615 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:51:48.615 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 5 --fsid=7f0bc282-eebc-452f-ae5f-6537699866e7 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/5 --osd-journal=td/osd-backfill-recovery-log/5/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76608/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:51:48.615 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:51:48.615 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:51:48.638 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:51:48.634+0000 7f0475e60ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:48.639 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:51:48.636+0000 7f0475e60ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:48.640 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:51:48.638+0000 7f0475e60ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:48.989 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 5 2020-03-11T22:51:48.989 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:51:48.989 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=5 2020-03-11T22:51:48.989 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:51:48.989 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:51:48.989 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:51:48.990 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:51:48.990 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:51:48.990 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.5 up' 2020-03-11T22:51:48.990 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:51:49.155 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:51:49.152+0000 7f0475e60ec0 -1 Falling back to public interface 2020-03-11T22:51:49.348 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:51:49.921 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:51:49.918+0000 7f0475e60ec0 -1 osd.5 0 log_to_monitors {default=true} 2020-03-11T22:51:50.349 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:51:50.350 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:51:50.350 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:51:50.350 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:51:50.350 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.5 up' 2020-03-11T22:51:50.350 INFO:tasks.workunit.client.0.smithi059.stdout:1 2020-03-11T22:51:50.741 INFO:tasks.workunit.client.0.smithi059.stdout:osd.5 up in weight 1 up_from 29 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/78823,v1:127.0.0.1:6843/78823] [v2:127.0.0.1:6844/78823,v1:127.0.0.1:6845/78823] exists,up 4acde14e-f4ad-4c62-9aa9-bb310069c29b 2020-03-11T22:51:50.741 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:51:50.741 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:51:50.741 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:51:50.741 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:59: _common_test: create_pool test 1 1 2020-03-11T22:51:50.742 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test 1 1 2020-03-11T22:51:51.374 INFO:tasks.workunit.client.0.smithi059.stderr:pool 'test' already exists 2020-03-11T22:51:51.382 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:51:52.385 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: seq 1 150 2020-03-11T22:51:52.386 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:52.386 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-1 /etc/passwd 2020-03-11T22:51:53.390 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:53.391 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-2 /etc/passwd 2020-03-11T22:51:53.517 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:53.518 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-3 /etc/passwd 2020-03-11T22:51:53.655 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:53.655 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-4 /etc/passwd 2020-03-11T22:51:53.810 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:53.810 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-5 /etc/passwd 2020-03-11T22:51:53.927 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:53.927 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-6 /etc/passwd 2020-03-11T22:51:54.002 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:54.002 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-7 /etc/passwd 2020-03-11T22:51:54.048 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:54.048 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-8 /etc/passwd 2020-03-11T22:51:54.086 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:54.086 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-9 /etc/passwd 2020-03-11T22:51:54.168 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:54.169 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-10 /etc/passwd 2020-03-11T22:51:54.207 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:54.207 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-11 /etc/passwd 2020-03-11T22:51:54.327 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:54.327 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-12 /etc/passwd 2020-03-11T22:51:54.419 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:54.420 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-13 /etc/passwd 2020-03-11T22:51:54.465 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:54.465 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-14 /etc/passwd 2020-03-11T22:51:54.499 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:54.499 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-15 /etc/passwd 2020-03-11T22:51:54.571 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:54.571 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-16 /etc/passwd 2020-03-11T22:51:54.670 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:54.670 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-17 /etc/passwd 2020-03-11T22:51:54.707 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:54.707 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-18 /etc/passwd 2020-03-11T22:51:54.787 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:54.787 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-19 /etc/passwd 2020-03-11T22:51:54.920 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:54.921 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-20 /etc/passwd 2020-03-11T22:51:54.957 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:54.957 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-21 /etc/passwd 2020-03-11T22:51:54.990 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:54.991 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-22 /etc/passwd 2020-03-11T22:51:55.024 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:55.024 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-23 /etc/passwd 2020-03-11T22:51:55.124 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:55.124 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-24 /etc/passwd 2020-03-11T22:51:55.165 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:55.166 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-25 /etc/passwd 2020-03-11T22:51:55.297 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:55.297 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-26 /etc/passwd 2020-03-11T22:51:55.422 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:55.422 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-27 /etc/passwd 2020-03-11T22:51:55.466 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:55.466 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-28 /etc/passwd 2020-03-11T22:51:55.515 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:55.515 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-29 /etc/passwd 2020-03-11T22:51:55.549 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:55.549 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-30 /etc/passwd 2020-03-11T22:51:55.622 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:55.623 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-31 /etc/passwd 2020-03-11T22:51:55.662 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:55.662 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-32 /etc/passwd 2020-03-11T22:51:55.765 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:55.765 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-33 /etc/passwd 2020-03-11T22:51:55.848 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:55.848 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-34 /etc/passwd 2020-03-11T22:51:55.907 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:55.907 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-35 /etc/passwd 2020-03-11T22:51:55.954 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:55.954 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-36 /etc/passwd 2020-03-11T22:51:56.007 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:56.008 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-37 /etc/passwd 2020-03-11T22:51:56.082 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:56.083 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-38 /etc/passwd 2020-03-11T22:51:56.120 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:56.120 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-39 /etc/passwd 2020-03-11T22:51:56.242 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:56.243 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-40 /etc/passwd 2020-03-11T22:51:56.358 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:56.358 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-41 /etc/passwd 2020-03-11T22:51:56.397 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:56.398 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-42 /etc/passwd 2020-03-11T22:51:56.434 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:56.434 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-43 /etc/passwd 2020-03-11T22:51:56.466 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:56.466 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-44 /etc/passwd 2020-03-11T22:51:56.569 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:56.570 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-45 /etc/passwd 2020-03-11T22:51:56.614 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:56.614 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-46 /etc/passwd 2020-03-11T22:51:56.692 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:56.692 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-47 /etc/passwd 2020-03-11T22:51:56.767 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:56.768 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-48 /etc/passwd 2020-03-11T22:51:56.806 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:56.806 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-49 /etc/passwd 2020-03-11T22:51:56.866 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:56.867 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-50 /etc/passwd 2020-03-11T22:51:56.934 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:56.934 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-51 /etc/passwd 2020-03-11T22:51:57.018 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:57.018 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-52 /etc/passwd 2020-03-11T22:51:57.056 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:57.057 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-53 /etc/passwd 2020-03-11T22:51:57.168 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:57.169 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-54 /etc/passwd 2020-03-11T22:51:57.244 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:57.244 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-55 /etc/passwd 2020-03-11T22:51:57.336 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:57.336 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-56 /etc/passwd 2020-03-11T22:51:57.444 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:57.444 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-57 /etc/passwd 2020-03-11T22:51:57.483 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:57.484 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-58 /etc/passwd 2020-03-11T22:51:57.561 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:57.561 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-59 /etc/passwd 2020-03-11T22:51:57.601 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:57.601 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-60 /etc/passwd 2020-03-11T22:51:57.726 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:57.726 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-61 /etc/passwd 2020-03-11T22:51:57.771 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:57.771 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-62 /etc/passwd 2020-03-11T22:51:57.854 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:57.879 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-63 /etc/passwd 2020-03-11T22:51:57.892 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:57.892 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-64 /etc/passwd 2020-03-11T22:51:57.925 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:57.926 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-65 /etc/passwd 2020-03-11T22:51:58.037 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:58.037 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-66 /etc/passwd 2020-03-11T22:51:58.075 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:58.075 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-67 /etc/passwd 2020-03-11T22:51:58.154 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:58.154 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-68 /etc/passwd 2020-03-11T22:51:58.190 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:58.190 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-69 /etc/passwd 2020-03-11T22:51:58.322 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:58.322 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-70 /etc/passwd 2020-03-11T22:51:58.359 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:58.359 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-71 /etc/passwd 2020-03-11T22:51:58.392 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:58.392 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-72 /etc/passwd 2020-03-11T22:51:58.455 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:58.455 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-73 /etc/passwd 2020-03-11T22:51:58.498 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:58.499 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-74 /etc/passwd 2020-03-11T22:51:58.622 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:58.622 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-75 /etc/passwd 2020-03-11T22:51:58.659 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:58.659 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-76 /etc/passwd 2020-03-11T22:51:58.739 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:58.740 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-77 /etc/passwd 2020-03-11T22:51:58.776 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:58.776 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-78 /etc/passwd 2020-03-11T22:51:58.809 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:58.809 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-79 /etc/passwd 2020-03-11T22:51:58.868 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:58.868 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-80 /etc/passwd 2020-03-11T22:51:58.940 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:58.940 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-81 /etc/passwd 2020-03-11T22:51:59.015 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:59.015 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-82 /etc/passwd 2020-03-11T22:51:59.051 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:59.051 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-83 /etc/passwd 2020-03-11T22:51:59.157 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:59.157 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-84 /etc/passwd 2020-03-11T22:51:59.197 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:59.197 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-85 /etc/passwd 2020-03-11T22:51:59.291 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:59.291 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-86 /etc/passwd 2020-03-11T22:51:59.326 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:59.326 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-87 /etc/passwd 2020-03-11T22:51:59.429 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:59.429 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-88 /etc/passwd 2020-03-11T22:51:59.509 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:59.509 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-89 /etc/passwd 2020-03-11T22:51:59.547 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:59.547 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-90 /etc/passwd 2020-03-11T22:51:59.626 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:59.626 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-91 /etc/passwd 2020-03-11T22:51:59.664 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:59.664 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-92 /etc/passwd 2020-03-11T22:51:59.727 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:59.727 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-93 /etc/passwd 2020-03-11T22:51:59.760 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:59.760 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-94 /etc/passwd 2020-03-11T22:51:59.834 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:59.834 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-95 /etc/passwd 2020-03-11T22:51:59.998 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:51:59.999 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-96 /etc/passwd 2020-03-11T22:52:00.068 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:00.068 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-97 /etc/passwd 2020-03-11T22:52:00.152 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:00.152 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-98 /etc/passwd 2020-03-11T22:52:00.189 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:00.189 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-99 /etc/passwd 2020-03-11T22:52:00.225 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:00.225 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-100 /etc/passwd 2020-03-11T22:52:00.289 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:00.290 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-101 /etc/passwd 2020-03-11T22:52:00.418 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:00.418 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-102 /etc/passwd 2020-03-11T22:52:00.502 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:00.503 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-103 /etc/passwd 2020-03-11T22:52:00.611 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:00.611 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-104 /etc/passwd 2020-03-11T22:52:00.649 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:00.650 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-105 /etc/passwd 2020-03-11T22:52:00.685 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:00.685 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-106 /etc/passwd 2020-03-11T22:52:00.719 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:00.719 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-107 /etc/passwd 2020-03-11T22:52:00.794 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:00.794 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-108 /etc/passwd 2020-03-11T22:52:00.850 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:00.850 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-109 /etc/passwd 2020-03-11T22:52:00.979 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:00.979 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-110 /etc/passwd 2020-03-11T22:52:01.062 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:01.062 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-111 /etc/passwd 2020-03-11T22:52:01.131 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:01.132 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-112 /etc/passwd 2020-03-11T22:52:01.166 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:01.167 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-113 /etc/passwd 2020-03-11T22:52:01.246 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:01.246 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-114 /etc/passwd 2020-03-11T22:52:01.329 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:01.330 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-115 /etc/passwd 2020-03-11T22:52:01.409 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:01.409 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-116 /etc/passwd 2020-03-11T22:52:01.496 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:01.497 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-117 /etc/passwd 2020-03-11T22:52:01.588 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:01.589 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-118 /etc/passwd 2020-03-11T22:52:01.628 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:01.628 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-119 /etc/passwd 2020-03-11T22:52:01.700 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:01.701 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-120 /etc/passwd 2020-03-11T22:52:01.736 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:01.736 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-121 /etc/passwd 2020-03-11T22:52:01.856 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:01.856 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-122 /etc/passwd 2020-03-11T22:52:01.894 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:01.895 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-123 /etc/passwd 2020-03-11T22:52:02.006 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:02.007 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-124 /etc/passwd 2020-03-11T22:52:02.090 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:02.090 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-125 /etc/passwd 2020-03-11T22:52:02.128 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:02.128 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-126 /etc/passwd 2020-03-11T22:52:02.161 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:02.161 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-127 /etc/passwd 2020-03-11T22:52:02.195 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:02.195 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-128 /etc/passwd 2020-03-11T22:52:02.324 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:02.324 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-129 /etc/passwd 2020-03-11T22:52:02.361 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:02.361 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-130 /etc/passwd 2020-03-11T22:52:02.433 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:02.433 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-131 /etc/passwd 2020-03-11T22:52:02.550 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:02.550 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-132 /etc/passwd 2020-03-11T22:52:02.587 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:02.587 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-133 /etc/passwd 2020-03-11T22:52:02.620 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:02.620 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-134 /etc/passwd 2020-03-11T22:52:02.653 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:02.654 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-135 /etc/passwd 2020-03-11T22:52:02.725 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:02.725 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-136 /etc/passwd 2020-03-11T22:52:02.762 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:02.762 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-137 /etc/passwd 2020-03-11T22:52:02.867 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:02.867 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-138 /etc/passwd 2020-03-11T22:52:02.959 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:02.959 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-139 /etc/passwd 2020-03-11T22:52:02.995 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:02.995 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-140 /etc/passwd 2020-03-11T22:52:03.034 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:03.034 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-141 /etc/passwd 2020-03-11T22:52:03.099 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:03.099 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-142 /etc/passwd 2020-03-11T22:52:03.184 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:03.184 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-143 /etc/passwd 2020-03-11T22:52:03.251 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:03.252 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-144 /etc/passwd 2020-03-11T22:52:03.418 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:03.418 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-145 /etc/passwd 2020-03-11T22:52:03.494 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:03.494 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-146 /etc/passwd 2020-03-11T22:52:03.532 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:03.533 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-147 /etc/passwd 2020-03-11T22:52:03.568 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:03.568 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-148 /etc/passwd 2020-03-11T22:52:03.604 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:03.604 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-149 /etc/passwd 2020-03-11T22:52:03.710 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:52:03.711 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-150 /etc/passwd 2020-03-11T22:52:03.750 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:67: _common_test: ceph pg dump pgs --format=json 2020-03-11T22:52:03.751 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:67: _common_test: jq '.pg_stats[0].up[]' 2020-03-11T22:52:04.094 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2020-03-11T22:52:04.102 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:67: _common_test: ceph osd out 1 0 2 2020-03-11T22:52:04.647 INFO:tasks.workunit.client.0.smithi059.stderr:osd.1 is already out. osd.0 is already out. osd.2 is already out. 2020-03-11T22:52:04.655 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:68: _common_test: '[' 0 '!=' 0 ']' 2020-03-11T22:52:04.655 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:74: _common_test: sleep 1 2020-03-11T22:52:05.658 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:75: _common_test: wait_for_clean 2020-03-11T22:52:05.659 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:52:05.659 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:52:05.659 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:52:05.659 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:52:05.659 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:52:05.659 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:52:05.659 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:52:05.660 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:52:05.660 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:52:05.660 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:52:05.764 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:52:05.765 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:52:05.765 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:52:05.765 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:52:05.765 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:52:06.132 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:52:06.132 INFO:tasks.workunit.client.0.smithi059.stderr:1 2020-03-11T22:52:06.133 INFO:tasks.workunit.client.0.smithi059.stderr:2 2020-03-11T22:52:06.133 INFO:tasks.workunit.client.0.smithi059.stderr:3 2020-03-11T22:52:06.133 INFO:tasks.workunit.client.0.smithi059.stderr:4 2020-03-11T22:52:06.133 INFO:tasks.workunit.client.0.smithi059.stderr:5' 2020-03-11T22:52:06.133 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:52:06.133 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:52:06.134 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:52:06.263 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836492 2020-03-11T22:52:06.263 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836492' 2020-03-11T22:52:06.263 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:52:06.263 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:52:06.394 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672971 2020-03-11T22:52:06.394 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836492 1-42949672971' 2020-03-11T22:52:06.395 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:52:06.395 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:52:06.524 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509449 2020-03-11T22:52:06.524 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836492 1-42949672971 2-64424509449' 2020-03-11T22:52:06.524 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:52:06.524 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:52:06.654 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=85899345928 2020-03-11T22:52:06.654 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836492 1-42949672971 2-64424509449 3-85899345928' 2020-03-11T22:52:06.654 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:52:06.655 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:52:06.783 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=107374182407 2020-03-11T22:52:06.784 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836492 1-42949672971 2-64424509449 3-85899345928 4-107374182407' 2020-03-11T22:52:06.784 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:52:06.784 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2020-03-11T22:52:06.914 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=124554051590 2020-03-11T22:52:06.914 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836492 1-42949672971 2-64424509449 3-85899345928 4-107374182407 5-124554051590' 2020-03-11T22:52:06.915 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:52:06.915 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836492 2020-03-11T22:52:06.915 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:52:06.916 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:52:06.917 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836492 2020-03-11T22:52:06.917 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:52:06.918 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836492 2020-03-11T22:52:06.918 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2020-03-11T22:52:06.918 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.0 seq 21474836492 2020-03-11T22:52:06.919 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:52:07.295 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836492 -lt 21474836492 2020-03-11T22:52:07.296 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:52:07.296 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672971 2020-03-11T22:52:07.297 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:52:07.298 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:52:07.298 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672971 2020-03-11T22:52:07.299 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:52:07.299 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672971 2020-03-11T22:52:07.300 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672971' 2020-03-11T22:52:07.300 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.1 seq 42949672971 2020-03-11T22:52:07.300 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:52:07.677 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672971 -lt 42949672971 2020-03-11T22:52:07.678 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:52:07.678 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509449 2020-03-11T22:52:07.678 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:52:07.679 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:52:07.680 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509449 2020-03-11T22:52:07.681 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:52:07.681 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509449 2020-03-11T22:52:07.682 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2020-03-11T22:52:07.682 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.2 seq 64424509449 2020-03-11T22:52:07.682 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:52:08.044 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509450 -lt 64424509449 2020-03-11T22:52:08.044 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:52:08.045 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-85899345928 2020-03-11T22:52:08.045 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:52:08.046 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:52:08.047 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-85899345928 2020-03-11T22:52:08.047 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:52:08.048 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=85899345928 2020-03-11T22:52:08.048 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 85899345928' 2020-03-11T22:52:08.048 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.3 seq 85899345928 2020-03-11T22:52:08.049 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:52:08.421 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 85899345928 -lt 85899345928 2020-03-11T22:52:08.421 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:52:08.422 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-107374182407 2020-03-11T22:52:08.422 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:52:08.422 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:52:08.424 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-107374182407 2020-03-11T22:52:08.424 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:52:08.425 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=107374182407 2020-03-11T22:52:08.425 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 107374182407' 2020-03-11T22:52:08.425 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.4 seq 107374182407 2020-03-11T22:52:08.425 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:52:08.834 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 107374182407 -lt 107374182407 2020-03-11T22:52:08.834 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:52:08.834 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 5-124554051590 2020-03-11T22:52:08.835 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:52:08.835 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=5 2020-03-11T22:52:08.837 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 5-124554051590 2020-03-11T22:52:08.837 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:52:08.837 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=124554051590 2020-03-11T22:52:08.838 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.5 seq 124554051590' 2020-03-11T22:52:08.838 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.5 seq 124554051590 2020-03-11T22:52:08.838 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 5 2020-03-11T22:52:09.202 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 124554051590 -lt 124554051590 2020-03-11T22:52:09.203 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:52:09.203 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:09.203 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:09.766 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:52:09.766 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:09.767 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:09.767 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:09.767 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:09.768 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:09.768 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:09.768 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:10.134 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:52:10.135 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:10.135 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:10.135 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:10.613 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:52:10.614 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:52:10.614 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:52:10.614 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:52:10.614 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:52:10.614 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:10.714 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:10.714 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:10.715 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:10.715 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:10.715 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:10.716 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:10.716 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:10.716 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:11.087 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:52:11.088 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:11.089 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:11.089 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:11.567 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:52:11.567 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:52:11.567 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:11.567 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:11.567 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:11.567 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:11.568 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:11.568 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:11.568 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:12.060 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=14233 2020-03-11T22:52:12.060 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 14233 '!=' null 2020-03-11T22:52:12.060 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:12.060 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:52:12.060 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:12.161 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:12.161 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:12.162 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:12.162 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:12.162 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:12.162 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:12.162 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:12.163 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:12.516 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:52:12.517 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:12.517 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:12.517 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:12.993 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:52:12.993 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:52:12.993 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:12.993 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:12.994 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:12.994 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:12.994 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:12.994 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:12.995 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:13.469 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=11026 2020-03-11T22:52:13.470 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 11026 '!=' null 2020-03-11T22:52:13.470 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:13.470 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:52:13.470 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:13.570 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:13.570 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:13.571 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:13.572 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:13.572 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:13.572 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:13.572 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:13.572 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:14.130 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:52:14.131 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:14.131 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:14.131 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:14.606 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:52:14.606 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:52:14.606 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:14.606 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:14.606 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:14.606 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:14.607 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:14.607 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:14.607 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:15.093 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=11026 2020-03-11T22:52:15.093 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 11026 '!=' null 2020-03-11T22:52:15.093 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:15.093 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:52:15.094 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:15.193 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:15.194 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:15.195 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:15.195 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:15.195 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:15.195 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:15.195 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:15.195 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:15.547 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:52:15.548 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:15.549 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:15.549 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:16.024 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:52:16.024 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:52:16.024 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:16.024 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:16.025 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:16.025 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:16.025 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:16.025 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:16.026 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:16.520 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=28979 2020-03-11T22:52:16.520 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 28979 '!=' null 2020-03-11T22:52:16.520 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:16.520 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:52:16.520 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:16.620 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:16.620 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:16.621 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:16.622 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:16.622 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:16.622 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:16.622 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:16.622 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:16.996 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:52:16.996 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:16.997 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:16.997 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:17.470 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:52:17.470 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:52:17.471 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:17.471 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:17.471 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:17.471 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:17.471 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:17.472 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:17.472 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:17.949 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=24385 2020-03-11T22:52:17.949 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 24385 '!=' null 2020-03-11T22:52:17.949 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:17.949 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:52:17.949 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:18.049 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:18.050 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:18.051 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:18.051 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:18.051 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:18.051 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:18.051 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:18.052 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:18.428 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:52:18.429 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:18.429 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:18.429 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:18.906 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:52:18.907 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:52:18.907 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:18.907 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:18.907 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:18.907 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:18.907 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:18.908 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:18.908 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:19.454 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=37948 2020-03-11T22:52:19.454 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 37948 '!=' null 2020-03-11T22:52:19.455 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:19.455 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:52:19.455 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:19.555 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:19.555 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:19.556 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:19.556 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:19.556 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:19.557 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:19.557 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:19.557 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:19.912 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:52:19.913 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:19.913 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:19.913 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:20.383 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:52:20.384 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:52:20.384 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:20.384 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:20.384 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:20.384 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:20.384 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:20.385 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:20.385 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:20.863 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=37948 2020-03-11T22:52:20.863 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 37948 '!=' null 2020-03-11T22:52:20.864 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:20.864 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:52:20.864 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:20.964 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:20.964 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:20.965 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:20.965 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:20.965 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:20.965 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:20.966 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:20.966 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:21.319 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:52:21.320 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:21.320 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:21.320 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:21.796 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:52:21.797 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:52:21.797 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:21.797 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:21.797 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:21.797 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:21.797 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:21.798 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:21.798 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:22.270 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=31048 2020-03-11T22:52:22.271 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 31048 '!=' null 2020-03-11T22:52:22.271 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:22.271 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:52:22.271 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:22.371 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:22.371 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:22.373 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:22.373 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:22.373 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:22.373 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:22.373 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:22.374 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:22.730 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:52:22.731 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:22.732 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:22.732 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:23.226 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:52:23.226 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:52:23.227 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:23.227 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:23.227 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:23.227 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:23.227 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:23.228 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:23.228 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:23.722 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=31048 2020-03-11T22:52:23.722 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 31048 '!=' null 2020-03-11T22:52:23.722 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:23.723 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:52:23.723 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:23.823 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:23.823 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:23.824 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:23.824 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:23.824 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:23.824 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:23.825 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:23.825 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:24.178 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:52:24.179 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:24.179 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:24.179 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:24.658 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:52:24.658 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:52:24.658 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:24.658 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:24.658 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:24.658 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:24.659 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:24.659 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:24.659 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:25.143 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=31048 2020-03-11T22:52:25.143 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 31048 '!=' null 2020-03-11T22:52:25.143 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:25.143 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:52:25.143 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:25.243 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:25.243 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:25.245 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:25.245 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:25.245 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:25.245 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:25.245 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:25.245 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:25.600 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:52:25.600 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:25.601 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:25.601 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:26.085 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:52:26.085 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:52:26.085 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:26.086 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:26.086 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:26.086 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:26.086 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:26.087 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:26.087 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:26.561 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=46572 2020-03-11T22:52:26.561 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 46572 '!=' null 2020-03-11T22:52:26.561 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:26.562 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:52:26.562 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:26.662 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:26.662 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:26.663 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:26.663 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:26.663 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:26.663 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:26.664 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:26.664 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:27.030 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:52:27.031 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:27.032 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:27.032 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:27.510 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:52:27.510 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:52:27.511 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:27.511 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:27.511 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:27.511 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:27.511 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:27.512 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:27.512 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:27.986 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=29898 2020-03-11T22:52:27.986 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 29898 '!=' null 2020-03-11T22:52:27.987 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:27.987 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:52:27.987 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:28.087 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:28.088 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:28.089 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:28.089 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:28.090 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:28.090 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:28.090 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:28.090 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:28.443 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:52:28.444 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:28.444 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:28.444 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:28.914 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:52:28.914 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:52:28.914 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:28.914 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:28.914 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:28.914 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:28.915 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:28.915 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:28.915 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:29.386 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=45998 2020-03-11T22:52:29.387 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 45998 '!=' null 2020-03-11T22:52:29.387 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:29.387 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:52:29.387 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:29.487 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:29.487 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:29.488 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:29.488 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:29.489 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:29.489 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:29.489 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:29.489 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:29.842 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:52:29.843 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:29.844 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:29.844 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:30.318 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:52:30.318 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:52:30.318 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:30.319 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:30.319 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:30.319 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:30.319 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:30.319 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:30.320 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:30.792 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=45998 2020-03-11T22:52:30.792 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 45998 '!=' null 2020-03-11T22:52:30.793 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:30.793 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:52:30.793 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:30.893 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:30.893 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:30.894 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:30.894 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:30.894 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:30.894 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:30.895 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:30.895 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:31.249 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:52:31.250 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:31.250 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:31.250 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:31.735 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:52:31.735 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:52:31.735 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:31.735 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:31.735 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:31.736 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:31.736 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:31.736 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:31.736 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:32.213 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=31624 2020-03-11T22:52:32.213 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 31624 '!=' null 2020-03-11T22:52:32.213 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:32.214 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:52:32.214 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:32.314 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:32.314 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:32.315 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:32.315 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:32.315 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:32.315 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:32.316 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:32.316 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:32.672 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:52:32.673 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:32.674 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:32.674 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:33.151 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:52:33.151 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:52:33.151 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:33.151 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:33.151 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:33.151 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:33.152 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:33.152 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:33.152 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:33.663 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=31624 2020-03-11T22:52:33.663 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 31624 '!=' null 2020-03-11T22:52:33.663 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:33.663 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:52:33.663 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:33.763 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:33.763 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:33.764 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:33.765 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:33.765 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:33.765 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:33.765 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:33.765 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:34.125 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:52:34.126 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:34.126 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:34.126 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:34.600 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:52:34.601 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:52:34.601 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:34.601 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:34.601 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:34.601 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:34.602 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:34.602 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:34.602 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:35.075 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=31624 2020-03-11T22:52:35.075 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 31624 '!=' null 2020-03-11T22:52:35.076 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:35.076 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:52:35.076 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:35.176 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:35.176 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:35.177 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:35.177 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:35.177 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:35.177 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:35.177 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:35.178 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:35.560 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:52:35.561 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:35.561 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:35.561 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:36.036 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:52:36.036 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:52:36.036 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:52:36.037 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:77: _common_test: ceph pg dump pgs --format=json 2020-03-11T22:52:36.037 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:77: _common_test: jq '.pg_stats[0].up_primary' 2020-03-11T22:52:36.400 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2020-03-11T22:52:36.408 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:77: _common_test: newprimary=4 2020-03-11T22:52:36.408 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:78: _common_test: kill_daemons 2020-03-11T22:52:36.409 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:52:36.409 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:52:36.409 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:52:36.410 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:52:36.410 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:52:36.720 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:52:36.720 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:80: _common_test: ERRORS=0 2020-03-11T22:52:36.720 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:81: _common_test: _objectstore_tool_nodown td/osd-backfill-recovery-log 4 --no-mon-config --pgid 1.0 --op log 2020-03-11T22:52:36.721 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:81: _common_test: tee td/osd-backfill-recovery-log/result.log 2020-03-11T22:52:36.721 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/osd-backfill-recovery-log 2020-03-11T22:52:36.721 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-03-11T22:52:36.721 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=4 2020-03-11T22:52:36.721 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-03-11T22:52:36.721 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/osd-backfill-recovery-log/4 2020-03-11T22:52:36.721 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-backfill-recovery-log/4 --no-mon-config --pgid 1.0 --op log 2020-03-11T22:52:37.857 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2020-03-11T22:52:37.857 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_log_t": { 2020-03-11T22:52:37.857 INFO:tasks.workunit.client.0.smithi059.stdout: "head": "33'150", 2020-03-11T22:52:37.858 INFO:tasks.workunit.client.0.smithi059.stdout: "tail": "33'148", 2020-03-11T22:52:37.858 INFO:tasks.workunit.client.0.smithi059.stdout: "log": [ 2020-03-11T22:52:37.858 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:52:37.858 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "modify", 2020-03-11T22:52:37.858 INFO:tasks.workunit.client.0.smithi059.stdout: "object": "1:e9c179f9:::obj-149:head", 2020-03-11T22:52:37.858 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "33'149", 2020-03-11T22:52:37.858 INFO:tasks.workunit.client.0.smithi059.stdout: "prior_version": "0'0", 2020-03-11T22:52:37.859 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4631.0:1", 2020-03-11T22:52:37.859 INFO:tasks.workunit.client.0.smithi059.stdout: "extra_reqids": [], 2020-03-11T22:52:37.859 INFO:tasks.workunit.client.0.smithi059.stdout: "mtime": "2020-03-11T22:52:03.624234+0000", 2020-03-11T22:52:37.859 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": 0, 2020-03-11T22:52:37.859 INFO:tasks.workunit.client.0.smithi059.stdout: "mod_desc": { 2020-03-11T22:52:37.859 INFO:tasks.workunit.client.0.smithi059.stdout: "object_mod_desc": { 2020-03-11T22:52:37.859 INFO:tasks.workunit.client.0.smithi059.stdout: "can_local_rollback": false, 2020-03-11T22:52:37.859 INFO:tasks.workunit.client.0.smithi059.stdout: "rollback_info_completed": false, 2020-03-11T22:52:37.860 INFO:tasks.workunit.client.0.smithi059.stdout: "ops": [] 2020-03-11T22:52:37.860 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:52:37.860 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:52:37.860 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_regions": { 2020-03-11T22:52:37.860 INFO:tasks.workunit.client.0.smithi059.stdout: "object_clean_regions": { 2020-03-11T22:52:37.860 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_offsets": "[6900~18446744073709544715]", 2020-03-11T22:52:37.860 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_omap": true, 2020-03-11T22:52:37.861 INFO:tasks.workunit.client.0.smithi059.stdout: "new_object": false 2020-03-11T22:52:37.861 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:52:37.861 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:52:37.861 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:52:37.861 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:52:37.861 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "modify", 2020-03-11T22:52:37.861 INFO:tasks.workunit.client.0.smithi059.stdout: "object": "1:cb39fd60:::obj-150:head", 2020-03-11T22:52:37.861 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "33'150", 2020-03-11T22:52:37.862 INFO:tasks.workunit.client.0.smithi059.stdout: "prior_version": "0'0", 2020-03-11T22:52:37.862 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4634.0:1", 2020-03-11T22:52:37.862 INFO:tasks.workunit.client.0.smithi059.stdout: "extra_reqids": [], 2020-03-11T22:52:37.862 INFO:tasks.workunit.client.0.smithi059.stdout: "mtime": "2020-03-11T22:52:03.731110+0000", 2020-03-11T22:52:37.862 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": 0, 2020-03-11T22:52:37.862 INFO:tasks.workunit.client.0.smithi059.stdout: "mod_desc": { 2020-03-11T22:52:37.862 INFO:tasks.workunit.client.0.smithi059.stdout: "object_mod_desc": { 2020-03-11T22:52:37.863 INFO:tasks.workunit.client.0.smithi059.stdout: "can_local_rollback": false, 2020-03-11T22:52:37.863 INFO:tasks.workunit.client.0.smithi059.stdout: "rollback_info_completed": false, 2020-03-11T22:52:37.863 INFO:tasks.workunit.client.0.smithi059.stdout: "ops": [] 2020-03-11T22:52:37.863 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:52:37.863 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:52:37.863 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_regions": { 2020-03-11T22:52:37.863 INFO:tasks.workunit.client.0.smithi059.stdout: "object_clean_regions": { 2020-03-11T22:52:37.863 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_offsets": "[6900~18446744073709544715]", 2020-03-11T22:52:37.864 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_omap": true, 2020-03-11T22:52:37.864 INFO:tasks.workunit.client.0.smithi059.stdout: "new_object": false 2020-03-11T22:52:37.864 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:52:37.864 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:52:37.864 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:52:37.864 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:52:37.864 INFO:tasks.workunit.client.0.smithi059.stdout: "dups": [ 2020-03-11T22:52:37.864 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:52:37.864 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4607.0:1", 2020-03-11T22:52:37.865 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "33'141", 2020-03-11T22:52:37.865 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "141", 2020-03-11T22:52:37.865 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:52:37.865 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:52:37.865 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:52:37.865 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4610.0:1", 2020-03-11T22:52:37.865 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "33'142", 2020-03-11T22:52:37.865 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "142", 2020-03-11T22:52:37.866 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:52:37.866 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:52:37.866 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:52:37.866 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4613.0:1", 2020-03-11T22:52:37.866 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "33'143", 2020-03-11T22:52:37.866 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "143", 2020-03-11T22:52:37.866 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:52:37.867 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:52:37.867 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:52:37.867 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4616.0:1", 2020-03-11T22:52:37.867 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "33'144", 2020-03-11T22:52:37.867 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "144", 2020-03-11T22:52:37.867 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:52:37.867 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:52:37.867 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:52:37.868 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4619.0:1", 2020-03-11T22:52:37.868 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "33'145", 2020-03-11T22:52:37.868 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "145", 2020-03-11T22:52:37.868 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:52:37.868 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:52:37.868 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:52:37.868 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4622.0:1", 2020-03-11T22:52:37.869 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "33'146", 2020-03-11T22:52:37.869 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "146", 2020-03-11T22:52:37.869 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:52:37.869 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:52:37.869 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:52:37.869 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4625.0:1", 2020-03-11T22:52:37.869 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "33'147", 2020-03-11T22:52:37.869 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "147", 2020-03-11T22:52:37.870 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:52:37.870 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:52:37.870 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:52:37.870 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4628.0:1", 2020-03-11T22:52:37.870 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "33'148", 2020-03-11T22:52:37.870 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "148", 2020-03-11T22:52:37.870 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:52:37.871 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:52:37.871 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:52:37.871 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:52:37.871 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_missing_t": { 2020-03-11T22:52:37.871 INFO:tasks.workunit.client.0.smithi059.stdout: "missing": [], 2020-03-11T22:52:37.871 INFO:tasks.workunit.client.0.smithi059.stdout: "may_include_deletes": true 2020-03-11T22:52:37.871 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:52:37.871 INFO:tasks.workunit.client.0.smithi059.stdout:} 2020-03-11T22:52:37.872 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:82: _common_test: jq '.pg_log_t.log | length' td/osd-backfill-recovery-log/result.log 2020-03-11T22:52:37.873 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:82: _common_test: LOGLEN=2 2020-03-11T22:52:37.873 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:83: _common_test: '[' 2 '!=' 2 ']' 2020-03-11T22:52:37.873 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:87: _common_test: jq '.pg_log_t.dups | length' td/osd-backfill-recovery-log/result.log 2020-03-11T22:52:37.873 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:87: _common_test: DUPSLEN=8 2020-03-11T22:52:37.873 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:88: _common_test: '[' 8 '!=' 8 ']' 2020-03-11T22:52:37.873 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:92: _common_test: grep 'copy_up_to\|copy_after' td/osd-backfill-recovery-log/osd.0.log td/osd-backfill-recovery-log/osd.1.log td/osd-backfill-recovery-log/osd.2.log td/osd-backfill-recovery-log/osd.3.log td/osd-backfill-recovery-log/osd.4.log td/osd-backfill-recovery-log/osd.5.log 2020-03-11T22:52:37.874 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:05.258+0000 7fa66841e700 20 copy_up_to max 2 2020-03-11T22:52:37.874 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:05.258+0000 7fa66841e700 20 copy_up_to copy log version 33'150 2020-03-11T22:52:37.874 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:05.258+0000 7fa66841e700 20 copy_up_to copy log version 33'149 2020-03-11T22:52:37.874 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:05.258+0000 7fa66841e700 20 copy_up_to/copy_after earliest_dup_version 141 2020-03-11T22:52:37.874 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:05.258+0000 7fa66841e700 20 copy_up_to/copy_after copy dup from log version 33'141 2020-03-11T22:52:37.874 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:05.258+0000 7fa66841e700 20 copy_up_to/copy_after copy dup from log version 33'142 2020-03-11T22:52:37.874 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:05.258+0000 7fa66841e700 20 copy_up_to/copy_after copy dup from log version 33'143 2020-03-11T22:52:37.875 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:05.258+0000 7fa66841e700 20 copy_up_to/copy_after copy dup from log version 33'144 2020-03-11T22:52:37.875 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:05.258+0000 7fa66841e700 20 copy_up_to/copy_after copy dup from log version 33'145 2020-03-11T22:52:37.875 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:05.258+0000 7fa66841e700 20 copy_up_to/copy_after copy dup from log version 33'146 2020-03-11T22:52:37.875 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:05.258+0000 7fa66841e700 20 copy_up_to/copy_after copy dup from log version 33'147 2020-03-11T22:52:37.875 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:05.258+0000 7fa66841e700 20 copy_up_to/copy_after copy dup from log version 33'148 2020-03-11T22:52:37.875 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:05.258+0000 7fa66841e700 20 copy_up_to max 2 2020-03-11T22:52:37.875 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:05.258+0000 7fa66841e700 20 copy_up_to copy log version 33'150 2020-03-11T22:52:37.876 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:05.258+0000 7fa66841e700 20 copy_up_to copy log version 33'149 2020-03-11T22:52:37.876 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:05.258+0000 7fa66841e700 20 copy_up_to/copy_after earliest_dup_version 141 2020-03-11T22:52:37.876 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:05.258+0000 7fa66841e700 20 copy_up_to/copy_after copy dup from log version 33'141 2020-03-11T22:52:37.876 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:05.258+0000 7fa66841e700 20 copy_up_to/copy_after copy dup from log version 33'142 2020-03-11T22:52:37.876 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:05.258+0000 7fa66841e700 20 copy_up_to/copy_after copy dup from log version 33'143 2020-03-11T22:52:37.876 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:05.258+0000 7fa66841e700 20 copy_up_to/copy_after copy dup from log version 33'144 2020-03-11T22:52:37.876 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:05.258+0000 7fa66841e700 20 copy_up_to/copy_after copy dup from log version 33'145 2020-03-11T22:52:37.876 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:05.258+0000 7fa66841e700 20 copy_up_to/copy_after copy dup from log version 33'146 2020-03-11T22:52:37.877 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:05.258+0000 7fa66841e700 20 copy_up_to/copy_after copy dup from log version 33'147 2020-03-11T22:52:37.877 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:05.258+0000 7fa66841e700 20 copy_up_to/copy_after copy dup from log version 33'148 2020-03-11T22:52:37.877 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:05.258+0000 7fa66841e700 20 copy_up_to max 2 2020-03-11T22:52:37.877 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:05.258+0000 7fa66841e700 20 copy_up_to copy log version 33'150 2020-03-11T22:52:37.877 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:05.258+0000 7fa66841e700 20 copy_up_to copy log version 33'149 2020-03-11T22:52:37.877 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:05.258+0000 7fa66841e700 20 copy_up_to/copy_after earliest_dup_version 141 2020-03-11T22:52:37.877 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:05.258+0000 7fa66841e700 20 copy_up_to/copy_after copy dup from log version 33'141 2020-03-11T22:52:37.878 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:05.258+0000 7fa66841e700 20 copy_up_to/copy_after copy dup from log version 33'142 2020-03-11T22:52:37.878 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:05.258+0000 7fa66841e700 20 copy_up_to/copy_after copy dup from log version 33'143 2020-03-11T22:52:37.884 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:05.258+0000 7fa66841e700 20 copy_up_to/copy_after copy dup from log version 33'144 2020-03-11T22:52:37.885 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:05.258+0000 7fa66841e700 20 copy_up_to/copy_after copy dup from log version 33'145 2020-03-11T22:52:37.885 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:05.258+0000 7fa66841e700 20 copy_up_to/copy_after copy dup from log version 33'146 2020-03-11T22:52:37.885 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:05.258+0000 7fa66841e700 20 copy_up_to/copy_after copy dup from log version 33'147 2020-03-11T22:52:37.885 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:05.258+0000 7fa66841e700 20 copy_up_to/copy_after copy dup from log version 33'148 2020-03-11T22:52:37.885 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:34.070+0000 7fa66841e700 20 copy_after v 33'150 2020-03-11T22:52:37.885 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:34.070+0000 7fa66841e700 20 copy_up_to/copy_after earliest_dup_version 141 2020-03-11T22:52:37.885 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:34.070+0000 7fa66841e700 20 copy_up_to/copy_after copy dup from log version 33'141 2020-03-11T22:52:37.886 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:34.070+0000 7fa66841e700 20 copy_up_to/copy_after copy dup from log version 33'142 2020-03-11T22:52:37.886 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:34.070+0000 7fa66841e700 20 copy_up_to/copy_after copy dup from log version 33'143 2020-03-11T22:52:37.886 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:34.070+0000 7fa66841e700 20 copy_up_to/copy_after copy dup from log version 33'144 2020-03-11T22:52:37.886 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:34.070+0000 7fa66841e700 20 copy_up_to/copy_after copy dup from log version 33'145 2020-03-11T22:52:37.886 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:34.070+0000 7fa66841e700 20 copy_up_to/copy_after copy dup from log version 33'146 2020-03-11T22:52:37.886 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:34.070+0000 7fa66841e700 20 copy_up_to/copy_after copy dup from log version 33'147 2020-03-11T22:52:37.886 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:34.070+0000 7fa66841e700 20 copy_up_to/copy_after copy dup from log version 33'148 2020-03-11T22:52:37.887 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:34.070+0000 7fa66841e700 20 copy_up_to/copy_after copy dup from log version 33'149 2020-03-11T22:52:37.887 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:52:34.070+0000 7fa66841e700 20 copy_up_to/copy_after copy dup from log version 33'150 2020-03-11T22:52:37.887 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:93: _common_test: rm -f td/osd-backfill-recovery-log/result.log 2020-03-11T22:52:37.888 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:94: _common_test: '[' 0 '!=' 0 ']' 2020-03-11T22:52:37.888 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:34: run: teardown td/osd-backfill-recovery-log 2020-03-11T22:52:37.888 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-recovery-log 2020-03-11T22:52:37.888 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:52:37.889 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-recovery-log KILL 2020-03-11T22:52:37.889 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:52:37.889 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:52:37.889 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:52:37.889 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:52:37.889 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:52:37.895 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:52:37.895 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:52:37.895 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:52:37.896 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:52:37.897 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:52:37.897 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:52:37.897 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:52:37.899 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:52:37.899 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:52:37.900 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:52:37.900 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:52:37.900 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:52:37.901 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:52:37.902 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-recovery-log 2020-03-11T22:52:37.919 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:52:37.919 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:52:37.920 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76608 2020-03-11T22:52:37.920 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76608 2020-03-11T22:52:37.921 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:52:37.922 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:52:37.922 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:31: run: for func in $funcs 2020-03-11T22:52:37.922 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:32: run: setup td/osd-backfill-recovery-log 2020-03-11T22:52:37.922 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-recovery-log 2020-03-11T22:52:37.922 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-recovery-log 2020-03-11T22:52:37.922 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-recovery-log 2020-03-11T22:52:37.923 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:52:37.923 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-recovery-log KILL 2020-03-11T22:52:37.923 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:52:37.923 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:52:37.923 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:52:37.923 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:52:37.923 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:52:37.924 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:52:37.924 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:52:37.925 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:52:37.925 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:52:37.925 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:52:37.925 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:52:37.926 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:52:37.927 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:52:37.927 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:52:37.928 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:52:37.928 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:52:37.928 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:52:37.929 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:52:37.929 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-recovery-log 2020-03-11T22:52:37.930 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:52:37.931 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:52:37.931 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76608 2020-03-11T22:52:37.931 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76608 2020-03-11T22:52:37.931 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:52:37.932 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:52:37.932 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-recovery-log 2020-03-11T22:52:37.933 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-11T22:52:37.933 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:52:37.933 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76608 2020-03-11T22:52:37.933 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76608 2020-03-11T22:52:37.936 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-11T22:52:37.936 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-03-11T22:52:37.936 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-11T22:52:37.936 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-recovery-log 1' TERM HUP INT 2020-03-11T22:52:37.936 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:33: run: TEST_backfill_log_2 td/osd-backfill-recovery-log 2020-03-11T22:52:37.937 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:111: TEST_backfill_log_2: local dir=td/osd-backfill-recovery-log 2020-03-11T22:52:37.937 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:113: TEST_backfill_log_2: _common_test td/osd-backfill-recovery-log '--osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2 148 150 2020-03-11T22:52:37.937 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:40: _common_test: local dir=td/osd-backfill-recovery-log 2020-03-11T22:52:37.937 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:41: _common_test: local 'extra_opts=--osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2020-03-11T22:52:37.937 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:42: _common_test: local loglen=2 2020-03-11T22:52:37.937 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:43: _common_test: local dupslen=148 2020-03-11T22:52:37.937 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:44: _common_test: local objects=150 2020-03-11T22:52:37.938 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:45: _common_test: local moreobjects=0 2020-03-11T22:52:37.938 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:47: _common_test: local OSDS=6 2020-03-11T22:52:37.938 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:49: _common_test: run_mon td/osd-backfill-recovery-log a 2020-03-11T22:52:37.938 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-recovery-log 2020-03-11T22:52:37.938 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-11T22:52:37.938 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-11T22:52:37.938 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-11T22:52:37.939 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-recovery-log/a 2020-03-11T22:52:37.939 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-recovery-log/a --run-dir=td/osd-backfill-recovery-log 2020-03-11T22:52:38.285 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-11T22:52:38.285 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:52:38.285 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:52:38.285 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:52:38.285 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:52:38.286 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76608 2020-03-11T22:52:38.286 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76608/$cluster-$name.asok' 2020-03-11T22:52:38.287 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-recovery-log/a '--log-file=td/osd-backfill-recovery-log/$name.log' '--admin-socket=/tmp/ceph-asok.76608/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-recovery-log/log --run-dir=td/osd-backfill-recovery-log '--pid-file=td/osd-backfill-recovery-log/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-11T22:52:38.478 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-11T22:52:38.478 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-11T22:52:38.479 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:52:38.479 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:52:38.479 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-11T22:52:38.480 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-11T22:52:38.480 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:52:38.480 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:52:38.481 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:52:38.481 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:52:38.481 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:52:38.481 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76608 2020-03-11T22:52:38.481 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76608/ceph-mon.a.asok 2020-03-11T22:52:38.481 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:52:38.482 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.76608/ceph-mon.a.asok config get fsid 2020-03-11T22:52:38.579 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-11T22:52:38.579 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:52:38.580 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:52:38.580 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-11T22:52:38.580 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-11T22:52:38.580 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:52:38.580 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:52:38.581 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:52:38.581 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:52:38.581 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:52:38.581 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76608 2020-03-11T22:52:38.582 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76608/ceph-mon.a.asok 2020-03-11T22:52:38.582 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:52:38.582 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.76608/ceph-mon.a.asok config get mon_host 2020-03-11T22:52:38.670 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:50: _common_test: run_mgr td/osd-backfill-recovery-log x 2020-03-11T22:52:38.670 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/osd-backfill-recovery-log 2020-03-11T22:52:38.671 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-11T22:52:38.671 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-11T22:52:38.671 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-11T22:52:38.671 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/osd-backfill-recovery-log/x 2020-03-11T22:52:38.672 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-11T22:52:39.596 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-11T22:52:39.596 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:52:39.597 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:52:39.597 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:52:39.597 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:52:39.597 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76608 2020-03-11T22:52:39.597 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76608/$cluster-$name.asok' 2020-03-11T22:52:39.598 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:52:39.598 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-recovery-log/x '--log-file=td/osd-backfill-recovery-log/$name.log' '--admin-socket=/tmp/ceph-asok.76608/$cluster-$name.asok' --run-dir=td/osd-backfill-recovery-log '--pid-file=td/osd-backfill-recovery-log/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:52:39.621 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:51: _common_test: export CEPH_ARGS 2020-03-11T22:52:39.622 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:52: _common_test: export 'EXTRA_OPTS= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2020-03-11T22:52:39.622 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:52: _common_test: EXTRA_OPTS=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2020-03-11T22:52:39.622 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: expr 6 - 1 2020-03-11T22:52:39.622 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: seq 0 5 2020-03-11T22:52:39.625 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:52:39.626 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 0 2020-03-11T22:52:39.626 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:52:39.626 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:52:39.626 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-11T22:52:39.626 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:52:39.626 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/0 2020-03-11T22:52:39.627 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=7f0bc282-eebc-452f-ae5f-6537699866e7 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:52:39.627 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:52:39.627 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:52:39.627 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:52:39.627 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/0' 2020-03-11T22:52:39.627 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/0/journal' 2020-03-11T22:52:39.627 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:52:39.628 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2020-03-11T22:52:39.628 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:52:39.628 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:52:39.628 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:52:39.628 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:52:39.628 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:52:39.628 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:52:39.629 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76608 2020-03-11T22:52:39.629 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76608/$cluster-$name.asok' 2020-03-11T22:52:39.629 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76608/$cluster-$name.asok' 2020-03-11T22:52:39.629 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:52:39.629 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:52:39.629 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:52:39.629 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:52:39.629 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:52:39.630 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:52:39.630 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:52:39.630 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:52:39.630 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:52:39.630 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:52:39.630 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/0 2020-03-11T22:52:39.630 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:52:39.631 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=979b2a02-855e-4cad-9271-520856b24726 2020-03-11T22:52:39.631 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 979b2a02-855e-4cad-9271-520856b24726' 2020-03-11T22:52:39.643 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:52:39.643 INFO:tasks.workunit.client.0.smithi059.stdout:add osd0 979b2a02-855e-4cad-9271-520856b24726 2020-03-11T22:52:39.644 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQC3a2ley3kHJhAADKtaQzyfaf3bNHtngqJzng== 2020-03-11T22:52:39.644 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQC3a2ley3kHJhAADKtaQzyfaf3bNHtngqJzng=="}' 2020-03-11T22:52:39.644 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 979b2a02-855e-4cad-9271-520856b24726 -i td/osd-backfill-recovery-log/0/new.json 2020-03-11T22:52:39.977 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:52:39.987 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/0/new.json 2020-03-11T22:52:39.988 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=7f0bc282-eebc-452f-ae5f-6537699866e7 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/0 --osd-journal=td/osd-backfill-recovery-log/0/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76608/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQC3a2ley3kHJhAADKtaQzyfaf3bNHtngqJzng== --osd-uuid 979b2a02-855e-4cad-9271-520856b24726 2020-03-11T22:52:40.012 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:52:40.008+0000 7fb615d9cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:40.014 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:52:40.010+0000 7fb615d9cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:40.016 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:52:40.012+0000 7fb615d9cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:40.016 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:52:40.012+0000 7fb615d9cec0 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2020-03-11T22:52:40.017 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:52:40.012+0000 7fb615d9cec0 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2020-03-11T22:52:40.017 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:52:40.012+0000 7fb615d9cec0 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2020-03-11T22:52:40.113 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:52:40.105+0000 7fb615d9cec0 -1 bluestore(td/osd-backfill-recovery-log/0) _read_fsid unparsable uuid 2020-03-11T22:52:43.226 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/0/keyring 2020-03-11T22:52:43.226 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:52:43.227 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-11T22:52:43.227 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:52:43.227 INFO:tasks.workunit.client.0.smithi059.stdout:adding osd0 key to auth repository 2020-03-11T22:52:43.720 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-11T22:52:43.720 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=7f0bc282-eebc-452f-ae5f-6537699866e7 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/0 --osd-journal=td/osd-backfill-recovery-log/0/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76608/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:52:43.720 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:52:43.720 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:52:43.721 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:52:43.721 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.0 2020-03-11T22:52:43.742 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:52:43.739+0000 7f0f2f568ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:43.744 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:52:43.741+0000 7f0f2f568ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:43.746 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:52:43.743+0000 7f0f2f568ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:44.096 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-11T22:52:44.096 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:52:44.097 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:52:44.097 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:52:44.097 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:52:44.097 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:52:44.097 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:52:44.097 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:52:44.097 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:52:44.098 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:52:44.262 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:52:44.259+0000 7f0f2f568ec0 -1 Falling back to public interface 2020-03-11T22:52:44.463 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:52:45.008 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:52:45.005+0000 7f0f2f568ec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-11T22:52:45.464 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:52:45.464 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:52:45.465 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:52:45.465 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:52:45.465 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:52:45.465 INFO:tasks.workunit.client.0.smithi059.stdout:1 2020-03-11T22:52:45.855 INFO:tasks.workunit.client.0.smithi059.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/85049,v1:127.0.0.1:6803/85049] [v2:127.0.0.1:6804/85049,v1:127.0.0.1:6805/85049] exists,up 979b2a02-855e-4cad-9271-520856b24726 2020-03-11T22:52:45.855 INFO:tasks.workunit.client.0.smithi059.stdout:add osd1 2dd7a8c9-1e18-4530-8fda-fca4224b05a0 2020-03-11T22:52:45.855 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:52:45.855 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:52:45.855 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:52:45.855 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:52:45.856 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 1 2020-03-11T22:52:45.856 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:52:45.856 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:52:45.856 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-11T22:52:45.856 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:52:45.856 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/1 2020-03-11T22:52:45.856 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=7f0bc282-eebc-452f-ae5f-6537699866e7 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:52:45.857 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:52:45.857 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:52:45.857 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:52:45.857 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/1' 2020-03-11T22:52:45.857 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/1/journal' 2020-03-11T22:52:45.857 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:52:45.857 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2020-03-11T22:52:45.858 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:52:45.858 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:52:45.858 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:52:45.858 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:52:45.858 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:52:45.858 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:52:45.858 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76608 2020-03-11T22:52:45.859 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76608/$cluster-$name.asok' 2020-03-11T22:52:45.859 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76608/$cluster-$name.asok' 2020-03-11T22:52:45.859 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:52:45.859 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:52:45.859 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:52:45.859 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:52:45.859 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:52:45.860 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:52:45.860 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:52:45.860 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:52:45.860 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:52:45.860 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:52:45.860 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/1 2020-03-11T22:52:45.860 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:52:45.861 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=2dd7a8c9-1e18-4530-8fda-fca4224b05a0 2020-03-11T22:52:45.861 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 2dd7a8c9-1e18-4530-8fda-fca4224b05a0' 2020-03-11T22:52:45.861 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:52:45.868 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQC9a2lezDedMxAAsu2l+2KOL4rVLYEHXbWlSA== 2020-03-11T22:52:45.869 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQC9a2lezDedMxAAsu2l+2KOL4rVLYEHXbWlSA=="}' 2020-03-11T22:52:45.869 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 2dd7a8c9-1e18-4530-8fda-fca4224b05a0 -i td/osd-backfill-recovery-log/1/new.json 2020-03-11T22:52:46.354 INFO:tasks.workunit.client.0.smithi059.stdout:1 2020-03-11T22:52:46.364 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/1/new.json 2020-03-11T22:52:46.365 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=7f0bc282-eebc-452f-ae5f-6537699866e7 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/1 --osd-journal=td/osd-backfill-recovery-log/1/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76608/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQC9a2lezDedMxAAsu2l+2KOL4rVLYEHXbWlSA== --osd-uuid 2dd7a8c9-1e18-4530-8fda-fca4224b05a0 2020-03-11T22:52:46.390 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:52:46.387+0000 7fc3027daec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:46.391 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:52:46.389+0000 7fc3027daec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:46.393 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:52:46.390+0000 7fc3027daec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:46.394 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:52:46.391+0000 7fc3027daec0 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2020-03-11T22:52:46.394 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:52:46.391+0000 7fc3027daec0 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2020-03-11T22:52:46.394 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:52:46.391+0000 7fc3027daec0 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2020-03-11T22:52:46.490 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:52:46.487+0000 7fc3027daec0 -1 bluestore(td/osd-backfill-recovery-log/1) _read_fsid unparsable uuid 2020-03-11T22:52:49.742 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/1/keyring 2020-03-11T22:52:49.742 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:52:49.743 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-03-11T22:52:49.744 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:52:49.744 INFO:tasks.workunit.client.0.smithi059.stdout:adding osd1 key to auth repository 2020-03-11T22:52:50.225 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.1 2020-03-11T22:52:50.226 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-03-11T22:52:50.226 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=7f0bc282-eebc-452f-ae5f-6537699866e7 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/1 --osd-journal=td/osd-backfill-recovery-log/1/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76608/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:52:50.226 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:52:50.226 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:52:50.226 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:52:50.247 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:52:50.245+0000 7f1605481ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:50.249 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:52:50.247+0000 7f1605481ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:50.250 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:52:50.247+0000 7f1605481ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:50.590 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-03-11T22:52:50.590 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:52:50.590 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:52:50.591 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:52:50.591 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:52:50.591 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:52:50.591 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:52:50.591 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:52:50.591 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:52:50.592 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:52:50.766 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:52:50.763+0000 7f1605481ec0 -1 Falling back to public interface 2020-03-11T22:52:50.958 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:52:51.556 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:52:51.553+0000 7f1605481ec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-11T22:52:51.959 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:52:51.960 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:52:51.960 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:52:51.960 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:52:51.960 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:52:51.960 INFO:tasks.workunit.client.0.smithi059.stdout:1 2020-03-11T22:52:52.372 INFO:tasks.workunit.client.0.smithi059.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/85404,v1:127.0.0.1:6811/85404] [v2:127.0.0.1:6812/85404,v1:127.0.0.1:6813/85404] exists,up 2dd7a8c9-1e18-4530-8fda-fca4224b05a0 2020-03-11T22:52:52.372 INFO:tasks.workunit.client.0.smithi059.stdout:add osd2 d3365f61-2841-477b-bcb1-98c0422857ce 2020-03-11T22:52:52.372 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:52:52.373 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:52:52.373 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:52:52.373 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:52:52.373 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 2 2020-03-11T22:52:52.373 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:52:52.373 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:52:52.373 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-03-11T22:52:52.374 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:52:52.374 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/2 2020-03-11T22:52:52.374 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=7f0bc282-eebc-452f-ae5f-6537699866e7 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:52:52.375 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:52:52.375 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:52:52.375 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:52:52.375 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/2' 2020-03-11T22:52:52.375 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/2/journal' 2020-03-11T22:52:52.375 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:52:52.375 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2020-03-11T22:52:52.376 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:52:52.376 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:52:52.376 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:52:52.376 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:52:52.376 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:52:52.377 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:52:52.377 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76608 2020-03-11T22:52:52.377 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76608/$cluster-$name.asok' 2020-03-11T22:52:52.377 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76608/$cluster-$name.asok' 2020-03-11T22:52:52.377 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:52:52.377 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:52:52.377 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:52:52.378 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:52:52.378 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:52:52.378 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:52:52.378 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:52:52.378 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:52:52.378 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:52:52.378 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:52:52.379 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/2 2020-03-11T22:52:52.379 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:52:52.379 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=d3365f61-2841-477b-bcb1-98c0422857ce 2020-03-11T22:52:52.379 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 d3365f61-2841-477b-bcb1-98c0422857ce' 2020-03-11T22:52:52.379 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:52:52.386 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQDEa2leERPYFhAAMFTVnJMzMJQfpK769WX7bg== 2020-03-11T22:52:52.386 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQDEa2leERPYFhAAMFTVnJMzMJQfpK769WX7bg=="}' 2020-03-11T22:52:52.386 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new d3365f61-2841-477b-bcb1-98c0422857ce -i td/osd-backfill-recovery-log/2/new.json 2020-03-11T22:52:52.983 INFO:tasks.workunit.client.0.smithi059.stdout:2 2020-03-11T22:52:52.993 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/2/new.json 2020-03-11T22:52:52.994 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=7f0bc282-eebc-452f-ae5f-6537699866e7 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/2 --osd-journal=td/osd-backfill-recovery-log/2/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76608/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDEa2leERPYFhAAMFTVnJMzMJQfpK769WX7bg== --osd-uuid d3365f61-2841-477b-bcb1-98c0422857ce 2020-03-11T22:52:53.018 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:52:53.015+0000 7f98034a9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:53.029 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:52:53.027+0000 7f98034a9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:53.031 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:52:53.029+0000 7f98034a9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:53.032 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:52:53.029+0000 7f98034a9ec0 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2020-03-11T22:52:53.032 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:52:53.029+0000 7f98034a9ec0 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2020-03-11T22:52:53.032 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:52:53.029+0000 7f98034a9ec0 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2020-03-11T22:52:53.117 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:52:53.114+0000 7f98034a9ec0 -1 bluestore(td/osd-backfill-recovery-log/2) _read_fsid unparsable uuid 2020-03-11T22:52:56.497 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/2/keyring 2020-03-11T22:52:56.497 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:52:56.498 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-03-11T22:52:56.498 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:52:56.498 INFO:tasks.workunit.client.0.smithi059.stdout:adding osd2 key to auth repository 2020-03-11T22:52:56.983 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-03-11T22:52:56.984 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=7f0bc282-eebc-452f-ae5f-6537699866e7 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/2 --osd-journal=td/osd-backfill-recovery-log/2/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76608/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:52:56.984 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:52:56.984 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:52:56.984 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:52:56.984 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.2 2020-03-11T22:52:57.006 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:52:57.003+0000 7f8c440d7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:57.007 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:52:57.005+0000 7f8c440d7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:57.009 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:52:57.006+0000 7f8c440d7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:57.350 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-03-11T22:52:57.350 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:52:57.350 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:52:57.350 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:52:57.350 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:52:57.351 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:52:57.351 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:52:57.351 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:52:57.351 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:52:57.351 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:52:57.528 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:52:57.525+0000 7f8c440d7ec0 -1 Falling back to public interface 2020-03-11T22:52:57.727 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:52:58.296 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:52:58.293+0000 7f8c440d7ec0 -1 osd.2 0 log_to_monitors {default=true} 2020-03-11T22:52:58.729 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:52:58.729 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:52:58.729 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:52:58.729 INFO:tasks.workunit.client.0.smithi059.stdout:1 2020-03-11T22:52:58.730 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:52:58.730 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:52:59.152 INFO:tasks.workunit.client.0.smithi059.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/85759,v1:127.0.0.1:6819/85759] [v2:127.0.0.1:6820/85759,v1:127.0.0.1:6821/85759] exists,up d3365f61-2841-477b-bcb1-98c0422857ce 2020-03-11T22:52:59.152 INFO:tasks.workunit.client.0.smithi059.stdout:add osd3 3c82e650-9fae-460f-9998-f449a6095c5f 2020-03-11T22:52:59.152 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:52:59.153 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:52:59.153 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:52:59.153 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:52:59.153 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 3 2020-03-11T22:52:59.153 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:52:59.153 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:52:59.154 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=3 2020-03-11T22:52:59.154 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:52:59.154 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/3 2020-03-11T22:52:59.154 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=7f0bc282-eebc-452f-ae5f-6537699866e7 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:52:59.154 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:52:59.154 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:52:59.154 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:52:59.155 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/3' 2020-03-11T22:52:59.155 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/3/journal' 2020-03-11T22:52:59.155 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:52:59.155 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2020-03-11T22:52:59.155 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:52:59.155 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:52:59.155 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:52:59.155 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:52:59.156 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:52:59.156 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:52:59.156 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76608 2020-03-11T22:52:59.156 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76608/$cluster-$name.asok' 2020-03-11T22:52:59.156 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76608/$cluster-$name.asok' 2020-03-11T22:52:59.156 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:52:59.156 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:52:59.157 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:52:59.157 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:52:59.157 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:52:59.157 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:52:59.157 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:52:59.157 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:52:59.157 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:52:59.158 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:52:59.158 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/3 2020-03-11T22:52:59.158 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:52:59.158 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=3c82e650-9fae-460f-9998-f449a6095c5f 2020-03-11T22:52:59.158 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd3 3c82e650-9fae-460f-9998-f449a6095c5f' 2020-03-11T22:52:59.158 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:52:59.166 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQDLa2leHHK7CRAAVQ2LqYqW4JphIxLdVGkbMw== 2020-03-11T22:52:59.166 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQDLa2leHHK7CRAAVQ2LqYqW4JphIxLdVGkbMw=="}' 2020-03-11T22:52:59.166 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 3c82e650-9fae-460f-9998-f449a6095c5f -i td/osd-backfill-recovery-log/3/new.json 2020-03-11T22:52:59.725 INFO:tasks.workunit.client.0.smithi059.stdout:3 2020-03-11T22:52:59.735 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/3/new.json 2020-03-11T22:52:59.736 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 3 --fsid=7f0bc282-eebc-452f-ae5f-6537699866e7 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/3 --osd-journal=td/osd-backfill-recovery-log/3/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76608/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDLa2leHHK7CRAAVQ2LqYqW4JphIxLdVGkbMw== --osd-uuid 3c82e650-9fae-460f-9998-f449a6095c5f 2020-03-11T22:52:59.761 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:52:59.758+0000 7f1574adbec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:59.787 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:52:59.784+0000 7f1574adbec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:59.789 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:52:59.786+0000 7f1574adbec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:59.789 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:52:59.786+0000 7f1574adbec0 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2020-03-11T22:52:59.790 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:52:59.786+0000 7f1574adbec0 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2020-03-11T22:52:59.790 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:52:59.786+0000 7f1574adbec0 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2020-03-11T22:52:59.895 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:52:59.892+0000 7f1574adbec0 -1 bluestore(td/osd-backfill-recovery-log/3) _read_fsid unparsable uuid 2020-03-11T22:53:03.004 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/3/keyring 2020-03-11T22:53:03.004 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:53:03.005 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd3 key to auth repository 2020-03-11T22:53:03.005 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:53:03.005 INFO:tasks.workunit.client.0.smithi059.stdout:adding osd3 key to auth repository 2020-03-11T22:53:03.514 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.3 2020-03-11T22:53:03.515 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 3 --fsid=7f0bc282-eebc-452f-ae5f-6537699866e7 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/3 --osd-journal=td/osd-backfill-recovery-log/3/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76608/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:53:03.515 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:53:03.515 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:53:03.515 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:53:03.515 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.3 2020-03-11T22:53:03.536 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:53:03.534+0000 7f8d556e1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:03.538 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:53:03.535+0000 7f8d556e1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:03.540 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:53:03.537+0000 7f8d556e1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:03.903 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 3 2020-03-11T22:53:03.903 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:53:03.903 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=3 2020-03-11T22:53:03.903 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:53:03.903 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:53:03.903 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:53:03.904 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:53:03.904 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:53:03.904 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:53:03.904 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:53:04.055 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:53:04.052+0000 7f8d556e1ec0 -1 Falling back to public interface 2020-03-11T22:53:04.267 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:53:04.820 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:53:04.817+0000 7f8d556e1ec0 -1 osd.3 0 log_to_monitors {default=true} 2020-03-11T22:53:05.268 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:53:05.269 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:53:05.269 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:53:05.269 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:53:05.269 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:53:05.269 INFO:tasks.workunit.client.0.smithi059.stdout:1 2020-03-11T22:53:05.669 INFO:tasks.workunit.client.0.smithi059.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/86114,v1:127.0.0.1:6827/86114] [v2:127.0.0.1:6828/86114,v1:127.0.0.1:6829/86114] exists,up 3c82e650-9fae-460f-9998-f449a6095c5f 2020-03-11T22:53:05.669 INFO:tasks.workunit.client.0.smithi059.stdout:add osd4 a530abf2-a37f-433d-a6fb-62f97397687e 2020-03-11T22:53:05.669 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:53:05.669 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:53:05.669 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:53:05.670 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:53:05.670 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 4 2020-03-11T22:53:05.670 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:53:05.670 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:53:05.670 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=4 2020-03-11T22:53:05.670 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:53:05.670 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/4 2020-03-11T22:53:05.671 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=7f0bc282-eebc-452f-ae5f-6537699866e7 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:53:05.671 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:53:05.671 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:53:05.671 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:53:05.671 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/4' 2020-03-11T22:53:05.671 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/4/journal' 2020-03-11T22:53:05.671 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:53:05.672 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2020-03-11T22:53:05.672 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:53:05.672 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:53:05.672 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:53:05.672 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:53:05.672 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:53:05.672 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:53:05.673 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76608 2020-03-11T22:53:05.673 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76608/$cluster-$name.asok' 2020-03-11T22:53:05.673 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76608/$cluster-$name.asok' 2020-03-11T22:53:05.673 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:53:05.673 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:53:05.673 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:53:05.673 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:53:05.674 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:53:05.674 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:53:05.674 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:53:05.674 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:53:05.674 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:53:05.675 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:53:05.675 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/4 2020-03-11T22:53:05.675 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:53:05.675 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=a530abf2-a37f-433d-a6fb-62f97397687e 2020-03-11T22:53:05.675 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd4 a530abf2-a37f-433d-a6fb-62f97397687e' 2020-03-11T22:53:05.676 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:53:05.682 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQDRa2lewauIKBAAPirudiX89PJ1rz/ixEXHVA== 2020-03-11T22:53:05.683 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQDRa2lewauIKBAAPirudiX89PJ1rz/ixEXHVA=="}' 2020-03-11T22:53:05.683 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new a530abf2-a37f-433d-a6fb-62f97397687e -i td/osd-backfill-recovery-log/4/new.json 2020-03-11T22:53:06.158 INFO:tasks.workunit.client.0.smithi059.stdout:4 2020-03-11T22:53:06.167 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/4/new.json 2020-03-11T22:53:06.168 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 4 --fsid=7f0bc282-eebc-452f-ae5f-6537699866e7 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/4 --osd-journal=td/osd-backfill-recovery-log/4/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76608/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDRa2lewauIKBAAPirudiX89PJ1rz/ixEXHVA== --osd-uuid a530abf2-a37f-433d-a6fb-62f97397687e 2020-03-11T22:53:06.193 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:53:06.190+0000 7fc549759ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:06.195 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:53:06.192+0000 7fc549759ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:06.197 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:53:06.194+0000 7fc549759ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:06.197 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:53:06.194+0000 7fc549759ec0 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2020-03-11T22:53:06.198 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:53:06.194+0000 7fc549759ec0 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2020-03-11T22:53:06.198 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:53:06.194+0000 7fc549759ec0 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2020-03-11T22:53:06.236 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:53:06.233+0000 7fc549759ec0 -1 bluestore(td/osd-backfill-recovery-log/4) _read_fsid unparsable uuid 2020-03-11T22:53:09.630 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/4/keyring 2020-03-11T22:53:09.630 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:53:09.631 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd4 key to auth repository 2020-03-11T22:53:09.631 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:53:09.631 INFO:tasks.workunit.client.0.smithi059.stdout:adding osd4 key to auth repository 2020-03-11T22:53:10.113 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.4 2020-03-11T22:53:10.113 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 4 --fsid=7f0bc282-eebc-452f-ae5f-6537699866e7 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/4 --osd-journal=td/osd-backfill-recovery-log/4/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76608/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:53:10.114 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:53:10.114 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:53:10.114 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:53:10.114 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.4 2020-03-11T22:53:10.135 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:53:10.132+0000 7fdc3a42aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:10.137 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:53:10.134+0000 7fdc3a42aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:10.139 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:53:10.136+0000 7fdc3a42aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:10.500 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 4 2020-03-11T22:53:10.500 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:53:10.500 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=4 2020-03-11T22:53:10.501 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:53:10.501 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:53:10.501 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:53:10.501 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:53:10.502 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:53:10.502 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.4 up' 2020-03-11T22:53:10.502 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:53:10.655 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:53:10.652+0000 7fdc3a42aec0 -1 Falling back to public interface 2020-03-11T22:53:10.860 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:53:11.412 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:53:11.409+0000 7fdc3a42aec0 -1 osd.4 0 log_to_monitors {default=true} 2020-03-11T22:53:11.861 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:53:11.862 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:53:11.862 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:53:11.862 INFO:tasks.workunit.client.0.smithi059.stdout:1 2020-03-11T22:53:11.862 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:53:11.863 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.4 up' 2020-03-11T22:53:12.264 INFO:tasks.workunit.client.0.smithi059.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/86469,v1:127.0.0.1:6835/86469] [v2:127.0.0.1:6836/86469,v1:127.0.0.1:6837/86469] exists,up a530abf2-a37f-433d-a6fb-62f97397687e 2020-03-11T22:53:12.264 INFO:tasks.workunit.client.0.smithi059.stdout:add osd5 1b3b5070-b371-4413-9dd3-a7649f9f4dfa 2020-03-11T22:53:12.264 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:53:12.264 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:53:12.265 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:53:12.265 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:53:12.265 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 5 2020-03-11T22:53:12.265 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:53:12.265 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:53:12.265 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=5 2020-03-11T22:53:12.266 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:53:12.266 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/5 2020-03-11T22:53:12.266 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=7f0bc282-eebc-452f-ae5f-6537699866e7 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:53:12.266 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:53:12.267 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:53:12.267 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:53:12.267 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/5' 2020-03-11T22:53:12.267 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/5/journal' 2020-03-11T22:53:12.267 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:53:12.267 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2020-03-11T22:53:12.267 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:53:12.268 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:53:12.268 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:53:12.268 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:53:12.268 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:53:12.268 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:53:12.268 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76608 2020-03-11T22:53:12.268 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76608/$cluster-$name.asok' 2020-03-11T22:53:12.269 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76608/$cluster-$name.asok' 2020-03-11T22:53:12.269 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:53:12.269 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:53:12.269 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:53:12.269 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:53:12.269 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:53:12.269 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:53:12.270 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:53:12.270 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:53:12.270 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:53:12.270 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:53:12.270 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/5 2020-03-11T22:53:12.270 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:53:12.271 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=1b3b5070-b371-4413-9dd3-a7649f9f4dfa 2020-03-11T22:53:12.271 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd5 1b3b5070-b371-4413-9dd3-a7649f9f4dfa' 2020-03-11T22:53:12.271 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:53:12.277 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQDYa2leFAlhEBAAs4986uFRmO+WZ6pa0I/k5A== 2020-03-11T22:53:12.278 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQDYa2leFAlhEBAAs4986uFRmO+WZ6pa0I/k5A=="}' 2020-03-11T22:53:12.278 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 1b3b5070-b371-4413-9dd3-a7649f9f4dfa -i td/osd-backfill-recovery-log/5/new.json 2020-03-11T22:53:12.868 INFO:tasks.workunit.client.0.smithi059.stdout:5 2020-03-11T22:53:12.878 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/5/new.json 2020-03-11T22:53:12.879 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 5 --fsid=7f0bc282-eebc-452f-ae5f-6537699866e7 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/5 --osd-journal=td/osd-backfill-recovery-log/5/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76608/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDYa2leFAlhEBAAs4986uFRmO+WZ6pa0I/k5A== --osd-uuid 1b3b5070-b371-4413-9dd3-a7649f9f4dfa 2020-03-11T22:53:12.903 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:53:12.900+0000 7f170a7eaec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:12.905 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:53:12.902+0000 7f170a7eaec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:12.906 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:53:12.903+0000 7f170a7eaec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:12.906 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:53:12.903+0000 7f170a7eaec0 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2020-03-11T22:53:12.906 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:53:12.903+0000 7f170a7eaec0 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2020-03-11T22:53:12.907 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:53:12.903+0000 7f170a7eaec0 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2020-03-11T22:53:13.003 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:53:13.000+0000 7f170a7eaec0 -1 bluestore(td/osd-backfill-recovery-log/5) _read_fsid unparsable uuid 2020-03-11T22:53:16.431 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/5/keyring 2020-03-11T22:53:16.431 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:53:16.432 INFO:tasks.workunit.client.0.smithi059.stdout:adding osd5 key to auth repository 2020-03-11T22:53:16.432 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd5 key to auth repository 2020-03-11T22:53:16.432 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:53:16.918 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.5 2020-03-11T22:53:16.918 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.5 2020-03-11T22:53:16.918 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 5 --fsid=7f0bc282-eebc-452f-ae5f-6537699866e7 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/5 --osd-journal=td/osd-backfill-recovery-log/5/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76608/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:53:16.919 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:53:16.919 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:53:16.919 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:53:16.939 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:53:16.937+0000 7fba38563ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:16.941 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:53:16.939+0000 7fba38563ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:16.942 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:53:16.939+0000 7fba38563ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:17.285 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 5 2020-03-11T22:53:17.286 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:53:17.286 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=5 2020-03-11T22:53:17.286 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:53:17.286 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:53:17.286 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:53:17.286 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:53:17.287 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:53:17.287 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.5 up' 2020-03-11T22:53:17.287 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:53:17.460 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:53:17.457+0000 7fba38563ec0 -1 Falling back to public interface 2020-03-11T22:53:17.678 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:53:18.222 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:53:18.219+0000 7fba38563ec0 -1 osd.5 0 log_to_monitors {default=true} 2020-03-11T22:53:18.680 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:53:18.680 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:53:18.680 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:53:18.680 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:53:18.680 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.5 up' 2020-03-11T22:53:18.681 INFO:tasks.workunit.client.0.smithi059.stdout:1 2020-03-11T22:53:19.072 INFO:tasks.workunit.client.0.smithi059.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/86824,v1:127.0.0.1:6843/86824] [v2:127.0.0.1:6844/86824,v1:127.0.0.1:6845/86824] exists,up 1b3b5070-b371-4413-9dd3-a7649f9f4dfa 2020-03-11T22:53:19.072 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:53:19.072 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:53:19.072 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:53:19.073 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:59: _common_test: create_pool test 1 1 2020-03-11T22:53:19.073 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test 1 1 2020-03-11T22:53:19.850 INFO:tasks.workunit.client.0.smithi059.stderr:pool 'test' already exists 2020-03-11T22:53:19.858 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:53:20.860 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: seq 1 150 2020-03-11T22:53:20.861 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:20.861 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-1 /etc/passwd 2020-03-11T22:53:20.907 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:20.908 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-2 /etc/passwd 2020-03-11T22:53:21.009 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:21.010 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-3 /etc/passwd 2020-03-11T22:53:21.227 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:21.227 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-4 /etc/passwd 2020-03-11T22:53:21.327 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:21.328 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-5 /etc/passwd 2020-03-11T22:53:21.366 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:21.366 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-6 /etc/passwd 2020-03-11T22:53:21.404 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:21.404 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-7 /etc/passwd 2020-03-11T22:53:21.522 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:21.522 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-8 /etc/passwd 2020-03-11T22:53:21.564 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:21.564 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-9 /etc/passwd 2020-03-11T22:53:21.695 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:21.695 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-10 /etc/passwd 2020-03-11T22:53:21.795 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:21.796 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-11 /etc/passwd 2020-03-11T22:53:21.832 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:21.833 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-12 /etc/passwd 2020-03-11T22:53:21.872 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:21.873 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-13 /etc/passwd 2020-03-11T22:53:21.914 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:21.914 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-14 /etc/passwd 2020-03-11T22:53:21.955 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:21.956 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-15 /etc/passwd 2020-03-11T22:53:22.024 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:22.024 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-16 /etc/passwd 2020-03-11T22:53:22.121 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:22.121 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-17 /etc/passwd 2020-03-11T22:53:22.204 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:22.204 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-18 /etc/passwd 2020-03-11T22:53:22.241 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:22.241 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-19 /etc/passwd 2020-03-11T22:53:22.308 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:22.308 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-20 /etc/passwd 2020-03-11T22:53:22.347 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:22.347 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-21 /etc/passwd 2020-03-11T22:53:22.413 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:22.413 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-22 /etc/passwd 2020-03-11T22:53:22.449 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:22.450 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-23 /etc/passwd 2020-03-11T22:53:22.580 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:22.580 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-24 /etc/passwd 2020-03-11T22:53:22.672 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:22.672 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-25 /etc/passwd 2020-03-11T22:53:22.708 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:22.708 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-26 /etc/passwd 2020-03-11T22:53:22.747 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:22.747 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-27 /etc/passwd 2020-03-11T22:53:22.789 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:22.789 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-28 /etc/passwd 2020-03-11T22:53:22.831 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:22.831 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-29 /etc/passwd 2020-03-11T22:53:22.905 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:22.905 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-30 /etc/passwd 2020-03-11T22:53:23.006 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:23.006 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-31 /etc/passwd 2020-03-11T22:53:23.090 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:23.090 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-32 /etc/passwd 2020-03-11T22:53:23.198 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:23.199 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-33 /etc/passwd 2020-03-11T22:53:23.274 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:23.274 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-34 /etc/passwd 2020-03-11T22:53:23.316 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:23.317 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-35 /etc/passwd 2020-03-11T22:53:23.356 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:23.356 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-36 /etc/passwd 2020-03-11T22:53:23.398 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:23.398 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-37 /etc/passwd 2020-03-11T22:53:23.532 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:23.533 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-38 /etc/passwd 2020-03-11T22:53:23.616 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:23.616 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-39 /etc/passwd 2020-03-11T22:53:23.655 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:23.655 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-40 /etc/passwd 2020-03-11T22:53:23.723 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:23.723 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-41 /etc/passwd 2020-03-11T22:53:23.775 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:23.775 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-42 /etc/passwd 2020-03-11T22:53:23.809 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:23.809 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-43 /etc/passwd 2020-03-11T22:53:23.842 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:23.842 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-44 /etc/passwd 2020-03-11T22:53:23.950 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:23.950 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-45 /etc/passwd 2020-03-11T22:53:24.076 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:24.076 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-46 /etc/passwd 2020-03-11T22:53:24.159 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:24.160 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-47 /etc/passwd 2020-03-11T22:53:24.198 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:24.199 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-48 /etc/passwd 2020-03-11T22:53:24.234 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:24.234 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-49 /etc/passwd 2020-03-11T22:53:24.314 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:24.314 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-50 /etc/passwd 2020-03-11T22:53:24.349 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:24.349 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-51 /etc/passwd 2020-03-11T22:53:24.451 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:24.452 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-52 /etc/passwd 2020-03-11T22:53:24.535 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:24.535 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-53 /etc/passwd 2020-03-11T22:53:24.584 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:24.584 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-54 /etc/passwd 2020-03-11T22:53:24.617 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:24.617 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-55 /etc/passwd 2020-03-11T22:53:24.693 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:24.693 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-56 /etc/passwd 2020-03-11T22:53:24.732 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:24.732 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-57 /etc/passwd 2020-03-11T22:53:24.767 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:24.767 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-58 /etc/passwd 2020-03-11T22:53:24.903 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:24.903 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-59 /etc/passwd 2020-03-11T22:53:24.986 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:24.987 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-60 /etc/passwd 2020-03-11T22:53:25.026 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:25.026 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-61 /etc/passwd 2020-03-11T22:53:25.059 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:25.059 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-62 /etc/passwd 2020-03-11T22:53:25.092 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:25.093 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-63 /etc/passwd 2020-03-11T22:53:25.199 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:25.199 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-64 /etc/passwd 2020-03-11T22:53:25.234 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:25.234 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-65 /etc/passwd 2020-03-11T22:53:25.337 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:25.337 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-66 /etc/passwd 2020-03-11T22:53:25.496 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:25.496 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-67 /etc/passwd 2020-03-11T22:53:25.534 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:25.535 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-68 /etc/passwd 2020-03-11T22:53:25.568 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:25.568 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-69 /etc/passwd 2020-03-11T22:53:25.601 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:25.602 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-70 /etc/passwd 2020-03-11T22:53:25.634 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:25.635 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-71 /etc/passwd 2020-03-11T22:53:25.668 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:25.668 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-72 /etc/passwd 2020-03-11T22:53:25.813 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:25.814 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-73 /etc/passwd 2020-03-11T22:53:25.897 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:25.897 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-74 /etc/passwd 2020-03-11T22:53:25.944 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:25.944 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-75 /etc/passwd 2020-03-11T22:53:25.968 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:25.968 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-76 /etc/passwd 2020-03-11T22:53:26.026 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:26.027 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-77 /etc/passwd 2020-03-11T22:53:26.060 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:26.060 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-78 /etc/passwd 2020-03-11T22:53:26.093 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:26.093 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-79 /etc/passwd 2020-03-11T22:53:26.127 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:26.127 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-80 /etc/passwd 2020-03-11T22:53:26.239 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:26.240 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-81 /etc/passwd 2020-03-11T22:53:26.323 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:26.323 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-82 /etc/passwd 2020-03-11T22:53:26.366 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:26.367 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-83 /etc/passwd 2020-03-11T22:53:26.402 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:26.402 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-84 /etc/passwd 2020-03-11T22:53:26.435 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:26.435 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-85 /etc/passwd 2020-03-11T22:53:26.469 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:26.469 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-86 /etc/passwd 2020-03-11T22:53:26.502 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:26.503 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-87 /etc/passwd 2020-03-11T22:53:26.640 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:26.640 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-88 /etc/passwd 2020-03-11T22:53:26.677 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:26.678 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-89 /etc/passwd 2020-03-11T22:53:26.710 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:26.711 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-90 /etc/passwd 2020-03-11T22:53:26.744 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:26.744 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-91 /etc/passwd 2020-03-11T22:53:26.777 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:26.778 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-92 /etc/passwd 2020-03-11T22:53:26.811 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:26.811 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-93 /etc/passwd 2020-03-11T22:53:26.886 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:26.886 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-94 /etc/passwd 2020-03-11T22:53:26.990 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:26.991 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-95 /etc/passwd 2020-03-11T22:53:27.027 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:27.027 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-96 /etc/passwd 2020-03-11T22:53:27.061 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:27.061 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-97 /etc/passwd 2020-03-11T22:53:27.094 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:27.094 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-98 /etc/passwd 2020-03-11T22:53:27.241 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:27.241 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-99 /etc/passwd 2020-03-11T22:53:27.277 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:27.278 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-100 /etc/passwd 2020-03-11T22:53:27.311 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:27.311 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-101 /etc/passwd 2020-03-11T22:53:27.466 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:27.467 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-102 /etc/passwd 2020-03-11T22:53:27.503 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:27.503 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-103 /etc/passwd 2020-03-11T22:53:27.536 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:27.537 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-104 /etc/passwd 2020-03-11T22:53:27.570 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:27.570 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-105 /etc/passwd 2020-03-11T22:53:27.603 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:27.603 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-106 /etc/passwd 2020-03-11T22:53:27.767 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:27.767 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-107 /etc/passwd 2020-03-11T22:53:27.803 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:27.803 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-108 /etc/passwd 2020-03-11T22:53:27.836 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:27.837 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-109 /etc/passwd 2020-03-11T22:53:27.870 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:27.870 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-110 /etc/passwd 2020-03-11T22:53:27.903 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:27.903 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-111 /etc/passwd 2020-03-11T22:53:27.938 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:27.938 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-112 /etc/passwd 2020-03-11T22:53:27.970 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:27.970 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-113 /etc/passwd 2020-03-11T22:53:28.092 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:28.093 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-114 /etc/passwd 2020-03-11T22:53:28.128 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:28.128 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-115 /etc/passwd 2020-03-11T22:53:28.162 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:28.162 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-116 /etc/passwd 2020-03-11T22:53:28.195 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:28.195 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-117 /etc/passwd 2020-03-11T22:53:28.228 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:28.228 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-118 /etc/passwd 2020-03-11T22:53:28.293 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:28.293 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-119 /etc/passwd 2020-03-11T22:53:28.360 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:28.360 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-120 /etc/passwd 2020-03-11T22:53:28.502 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:28.502 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-121 /etc/passwd 2020-03-11T22:53:28.541 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:28.541 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-122 /etc/passwd 2020-03-11T22:53:28.602 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:28.602 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-123 /etc/passwd 2020-03-11T22:53:28.637 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:28.638 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-124 /etc/passwd 2020-03-11T22:53:28.670 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:28.671 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-125 /etc/passwd 2020-03-11T22:53:28.704 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:28.704 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-126 /etc/passwd 2020-03-11T22:53:28.738 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:28.738 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-127 /etc/passwd 2020-03-11T22:53:28.894 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:28.895 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-128 /etc/passwd 2020-03-11T22:53:28.933 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:28.934 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-129 /etc/passwd 2020-03-11T22:53:29.020 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:29.020 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-130 /etc/passwd 2020-03-11T22:53:29.058 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:29.059 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-131 /etc/passwd 2020-03-11T22:53:29.094 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:29.094 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-132 /etc/passwd 2020-03-11T22:53:29.162 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:29.163 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-133 /etc/passwd 2020-03-11T22:53:29.237 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:29.237 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-134 /etc/passwd 2020-03-11T22:53:29.345 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:29.345 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-135 /etc/passwd 2020-03-11T22:53:29.471 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:29.471 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-136 /etc/passwd 2020-03-11T22:53:29.510 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:29.511 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-137 /etc/passwd 2020-03-11T22:53:29.546 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:29.546 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-138 /etc/passwd 2020-03-11T22:53:29.582 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:29.582 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-139 /etc/passwd 2020-03-11T22:53:29.612 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:29.613 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-140 /etc/passwd 2020-03-11T22:53:29.646 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:29.646 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-141 /etc/passwd 2020-03-11T22:53:29.797 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:29.797 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-142 /etc/passwd 2020-03-11T22:53:29.888 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:29.889 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-143 /etc/passwd 2020-03-11T22:53:29.927 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:29.928 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-144 /etc/passwd 2020-03-11T22:53:29.963 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:29.963 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-145 /etc/passwd 2020-03-11T22:53:30.027 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:30.027 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-146 /etc/passwd 2020-03-11T22:53:30.062 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:30.063 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-147 /etc/passwd 2020-03-11T22:53:30.096 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:30.096 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-148 /etc/passwd 2020-03-11T22:53:30.222 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:30.222 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-149 /etc/passwd 2020-03-11T22:53:30.356 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:30.357 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-150 /etc/passwd 2020-03-11T22:53:30.395 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:67: _common_test: ceph pg dump pgs --format=json 2020-03-11T22:53:30.395 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:67: _common_test: jq '.pg_stats[0].up[]' 2020-03-11T22:53:30.739 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2020-03-11T22:53:30.747 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:67: _common_test: ceph osd out 1 0 2 2020-03-11T22:53:31.300 INFO:tasks.workunit.client.0.smithi059.stderr:osd.1 is already out. osd.0 is already out. osd.2 is already out. 2020-03-11T22:53:31.308 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:68: _common_test: '[' 0 '!=' 0 ']' 2020-03-11T22:53:31.308 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:74: _common_test: sleep 1 2020-03-11T22:53:32.310 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:75: _common_test: wait_for_clean 2020-03-11T22:53:32.310 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:53:32.310 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:53:32.311 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:53:32.311 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:53:32.311 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:53:32.312 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:53:32.312 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:53:32.312 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:53:32.312 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:53:32.312 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:53:32.419 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:53:32.419 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:53:32.419 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:53:32.420 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:53:32.420 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:53:32.882 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:53:32.882 INFO:tasks.workunit.client.0.smithi059.stderr:1 2020-03-11T22:53:32.882 INFO:tasks.workunit.client.0.smithi059.stderr:2 2020-03-11T22:53:32.882 INFO:tasks.workunit.client.0.smithi059.stderr:3 2020-03-11T22:53:32.882 INFO:tasks.workunit.client.0.smithi059.stderr:4 2020-03-11T22:53:32.883 INFO:tasks.workunit.client.0.smithi059.stderr:5' 2020-03-11T22:53:32.883 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:53:32.883 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:53:32.883 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:53:33.078 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836492 2020-03-11T22:53:33.078 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836492' 2020-03-11T22:53:33.079 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:53:33.079 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:53:33.245 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672971 2020-03-11T22:53:33.246 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836492 1-42949672971' 2020-03-11T22:53:33.246 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:53:33.246 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:53:33.374 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509449 2020-03-11T22:53:33.375 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836492 1-42949672971 2-64424509449' 2020-03-11T22:53:33.375 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:53:33.375 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:53:33.503 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=85899345928 2020-03-11T22:53:33.503 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836492 1-42949672971 2-64424509449 3-85899345928' 2020-03-11T22:53:33.504 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:53:33.504 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:53:33.634 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=107374182407 2020-03-11T22:53:33.634 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836492 1-42949672971 2-64424509449 3-85899345928 4-107374182407' 2020-03-11T22:53:33.634 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:53:33.635 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2020-03-11T22:53:33.763 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=128849018885 2020-03-11T22:53:33.763 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836492 1-42949672971 2-64424509449 3-85899345928 4-107374182407 5-128849018885' 2020-03-11T22:53:33.764 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:53:33.764 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836492 2020-03-11T22:53:33.765 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:53:33.765 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:53:33.765 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836492 2020-03-11T22:53:33.766 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:53:33.766 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836492 2020-03-11T22:53:33.766 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2020-03-11T22:53:33.767 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:53:33.767 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.0 seq 21474836492 2020-03-11T22:53:34.153 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836491 -lt 21474836492 2020-03-11T22:53:34.153 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:53:35.154 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:53:35.155 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:53:35.556 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836492 -lt 21474836492 2020-03-11T22:53:35.556 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:53:35.556 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672971 2020-03-11T22:53:35.556 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:53:35.557 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:53:35.557 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672971 2020-03-11T22:53:35.557 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:53:35.558 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672971 2020-03-11T22:53:35.558 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672971' 2020-03-11T22:53:35.558 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:53:35.559 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.1 seq 42949672971 2020-03-11T22:53:35.921 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672971 -lt 42949672971 2020-03-11T22:53:35.921 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:53:35.922 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509449 2020-03-11T22:53:35.922 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:53:35.923 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:53:35.923 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509449 2020-03-11T22:53:35.923 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:53:35.924 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509449 2020-03-11T22:53:35.924 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2020-03-11T22:53:35.924 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:53:35.924 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.2 seq 64424509449 2020-03-11T22:53:36.291 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509450 -lt 64424509449 2020-03-11T22:53:36.291 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:53:36.292 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-85899345928 2020-03-11T22:53:36.292 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:53:36.293 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:53:36.293 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-85899345928 2020-03-11T22:53:36.293 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:53:36.294 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=85899345928 2020-03-11T22:53:36.294 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 85899345928' 2020-03-11T22:53:36.295 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:53:36.295 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.3 seq 85899345928 2020-03-11T22:53:36.677 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 85899345928 -lt 85899345928 2020-03-11T22:53:36.677 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:53:36.677 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-107374182407 2020-03-11T22:53:36.678 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:53:36.678 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:53:36.679 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-107374182407 2020-03-11T22:53:36.679 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:53:36.679 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=107374182407 2020-03-11T22:53:36.680 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 107374182407' 2020-03-11T22:53:36.680 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:53:36.680 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.4 seq 107374182407 2020-03-11T22:53:37.048 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 107374182407 -lt 107374182407 2020-03-11T22:53:37.048 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:53:37.049 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 5-128849018885 2020-03-11T22:53:37.049 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:53:37.049 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=5 2020-03-11T22:53:37.050 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 5-128849018885 2020-03-11T22:53:37.050 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:53:37.051 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=128849018885 2020-03-11T22:53:37.051 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.5 seq 128849018885' 2020-03-11T22:53:37.051 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 5 2020-03-11T22:53:37.051 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.5 seq 128849018885 2020-03-11T22:53:37.520 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 128849018886 -lt 128849018885 2020-03-11T22:53:37.521 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:53:37.521 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:37.522 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:38.022 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:53:38.022 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:38.022 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:38.022 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:38.022 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:38.022 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:38.023 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:38.023 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:38.375 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:53:38.375 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:38.376 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:38.376 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:38.845 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:53:38.845 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:53:38.845 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:53:38.846 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:53:38.846 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:53:38.846 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:38.946 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:38.946 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:38.947 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:38.947 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:38.947 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:38.947 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:38.948 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:38.948 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:39.335 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:53:39.336 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:39.336 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:39.336 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:39.814 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:53:39.814 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:53:39.814 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:39.814 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:39.814 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:39.815 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:39.815 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:39.815 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:39.815 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:40.281 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=13380 2020-03-11T22:53:40.281 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 13380 '!=' null 2020-03-11T22:53:40.281 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:40.282 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:53:40.282 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:40.383 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:40.383 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:40.383 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:40.383 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:40.384 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:40.384 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:40.384 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:40.384 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:40.736 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:53:40.736 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:40.736 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:40.737 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:41.371 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:53:41.371 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:53:41.372 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:41.372 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:41.372 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:41.372 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:41.372 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:41.372 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:41.372 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:41.845 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=10653 2020-03-11T22:53:41.845 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 10653 '!=' null 2020-03-11T22:53:41.846 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:41.846 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:53:41.846 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:41.946 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:41.946 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:41.947 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:41.947 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:41.947 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:41.948 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:41.948 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:41.948 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:42.304 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:53:42.305 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:42.305 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:42.305 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:42.776 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:53:42.777 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:53:42.777 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:42.777 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:42.777 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:42.777 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:42.777 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:42.778 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:42.778 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:43.296 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=27390 2020-03-11T22:53:43.296 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 27390 '!=' null 2020-03-11T22:53:43.296 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:43.296 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:53:43.297 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:43.397 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:43.398 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:43.398 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:43.398 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:43.398 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:43.398 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:43.398 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:43.399 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:43.750 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:53:43.750 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:43.750 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:43.751 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:44.219 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:53:44.220 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:53:44.220 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:44.220 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:44.220 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:44.220 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:44.220 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:44.220 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:44.221 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:44.699 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=27390 2020-03-11T22:53:44.699 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 27390 '!=' null 2020-03-11T22:53:44.700 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:44.700 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:53:44.700 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:44.800 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:44.800 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:44.801 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:44.801 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:44.801 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:44.802 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:44.802 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:44.802 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:45.151 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:53:45.152 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:45.152 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:45.152 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:45.627 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:53:45.627 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:53:45.627 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:45.627 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:45.628 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:45.628 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:45.628 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:45.628 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:45.628 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:46.100 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=22998 2020-03-11T22:53:46.100 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 22998 '!=' null 2020-03-11T22:53:46.101 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:46.101 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:53:46.101 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:46.201 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:46.201 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:46.202 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:46.202 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:46.203 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:46.203 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:46.203 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:46.203 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:46.556 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:53:46.557 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:46.557 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:46.557 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:47.030 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:53:47.030 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:53:47.030 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:47.030 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:47.030 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:47.030 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:47.031 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:47.031 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:47.031 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:47.535 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=38522 2020-03-11T22:53:47.535 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 38522 '!=' null 2020-03-11T22:53:47.535 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:47.536 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:53:47.536 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:47.636 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:47.637 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:47.637 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:47.637 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:47.637 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:47.637 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:47.637 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:47.638 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:47.989 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:53:47.989 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:47.989 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:47.989 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:48.464 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:53:48.464 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:53:48.465 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:48.465 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:48.465 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:48.465 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:48.466 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:48.466 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:48.466 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:48.944 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=38522 2020-03-11T22:53:48.944 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 38522 '!=' null 2020-03-11T22:53:48.945 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:48.945 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:53:48.945 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:49.044 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:49.045 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:49.046 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:49.046 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:49.046 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:49.046 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:49.046 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:49.046 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:49.402 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:53:49.402 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:49.403 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:49.403 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:49.897 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:53:49.898 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:53:49.898 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:49.898 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:49.898 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:49.899 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:49.899 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:49.899 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:49.899 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:50.372 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=31048 2020-03-11T22:53:50.372 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 31048 '!=' null 2020-03-11T22:53:50.372 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:50.373 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:53:50.373 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:50.473 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:50.481 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:50.481 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:50.481 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:50.481 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:50.481 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:50.481 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:50.482 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:50.829 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:53:50.829 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:50.830 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:50.830 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:51.306 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:53:51.306 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:53:51.306 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:51.306 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:51.307 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:51.307 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:51.307 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:51.307 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:51.307 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:51.800 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=31049 2020-03-11T22:53:51.800 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 31049 '!=' null 2020-03-11T22:53:51.800 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:51.800 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:53:51.800 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:51.901 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:51.901 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:51.902 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:51.902 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:51.902 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:51.902 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:51.902 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:51.903 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:52.252 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:53:52.252 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:52.253 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:52.253 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:52.732 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:53:52.732 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:53:52.732 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:52.732 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:52.733 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:52.733 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:52.733 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:52.733 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:52.733 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:53.215 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=31049 2020-03-11T22:53:53.215 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 31049 '!=' null 2020-03-11T22:53:53.215 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:53.215 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:53:53.215 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:53.316 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:53.316 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:53.316 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:53.317 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:53.317 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:53.317 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:53.317 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:53.317 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:53.670 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:53:53.670 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:53.671 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:53.671 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:54.139 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:53:54.139 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:53:54.139 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:54.139 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:54.139 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:54.140 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:54.140 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:54.140 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:54.140 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:54.615 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=46573 2020-03-11T22:53:54.615 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 46573 '!=' null 2020-03-11T22:53:54.615 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:54.615 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:53:54.616 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:54.716 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:54.716 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:54.716 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:54.717 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:54.717 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:54.717 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:54.717 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:54.717 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:55.070 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:53:55.070 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:55.071 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:55.071 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:55.556 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:53:55.556 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:53:55.557 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:55.557 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:55.557 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:55.557 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:55.557 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:55.557 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:55.558 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:56.053 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=31049 2020-03-11T22:53:56.053 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 31049 '!=' null 2020-03-11T22:53:56.054 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:56.054 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:53:56.054 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:56.155 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:56.155 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:56.155 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:56.155 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:56.155 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:56.155 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:56.156 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:56.156 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:56.508 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:53:56.508 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:56.508 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:56.509 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:56.980 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:53:56.980 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:53:56.980 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:56.981 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:56.981 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:56.981 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:56.981 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:56.981 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:56.981 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:57.457 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=46573 2020-03-11T22:53:57.458 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 46573 '!=' null 2020-03-11T22:53:57.458 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:57.458 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:53:57.458 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:57.559 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:57.559 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:57.560 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:57.560 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:57.560 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:57.560 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:57.561 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:57.561 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:57.911 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:53:57.911 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:57.912 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:57.912 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:58.397 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:53:58.397 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:53:58.397 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:58.397 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:58.397 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:58.397 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:58.398 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:58.398 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:58.398 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:58.875 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=46573 2020-03-11T22:53:58.875 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 46573 '!=' null 2020-03-11T22:53:58.875 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:58.875 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:53:58.876 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:58.975 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:58.976 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:58.976 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:58.976 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:58.976 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:58.976 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:58.976 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:58.977 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:59.335 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:53:59.335 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:59.336 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:59.336 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:59.809 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:53:59.809 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:53:59.809 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:59.810 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:59.810 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:59.810 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:59.810 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:59.810 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:59.810 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:00.287 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=31049 2020-03-11T22:54:00.288 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 31049 '!=' null 2020-03-11T22:54:00.288 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:00.288 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:54:00.288 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:00.388 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:00.389 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:00.389 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:00.389 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:00.389 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:00.389 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:00.389 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:00.390 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:00.746 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:00.746 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:00.747 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:00.747 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:01.221 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:54:01.221 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:01.222 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:01.222 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:01.222 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:01.222 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:01.223 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:01.223 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:01.223 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:01.715 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=31049 2020-03-11T22:54:01.716 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 31049 '!=' null 2020-03-11T22:54:01.716 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:01.716 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:54:01.717 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:01.816 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:01.816 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:01.817 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:01.817 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:01.817 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:01.818 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:01.818 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:01.818 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:02.254 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:02.254 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:02.255 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:02.255 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:02.733 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:54:02.733 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:02.733 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:02.734 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:02.734 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:02.734 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:02.734 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:02.734 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:02.734 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:03.233 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=31049 2020-03-11T22:54:03.234 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 31049 '!=' null 2020-03-11T22:54:03.234 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:03.234 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:54:03.234 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:03.334 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:03.334 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:03.335 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:03.335 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:03.335 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:03.335 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:03.335 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:03.335 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:03.681 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:54:03.682 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:03.682 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:03.682 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:04.163 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:54:04.163 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:54:04.164 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:54:04.164 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:77: _common_test: ceph pg dump pgs --format=json 2020-03-11T22:54:04.164 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:77: _common_test: jq '.pg_stats[0].up_primary' 2020-03-11T22:54:04.506 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2020-03-11T22:54:04.515 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:77: _common_test: newprimary=4 2020-03-11T22:54:04.515 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:78: _common_test: kill_daemons 2020-03-11T22:54:04.516 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:54:04.516 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:54:04.516 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:54:04.516 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:54:04.516 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:54:04.826 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:54:04.826 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:80: _common_test: ERRORS=0 2020-03-11T22:54:04.826 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:81: _common_test: _objectstore_tool_nodown td/osd-backfill-recovery-log 4 --no-mon-config --pgid 1.0 --op log 2020-03-11T22:54:04.826 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/osd-backfill-recovery-log 2020-03-11T22:54:04.826 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-03-11T22:54:04.827 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:81: _common_test: tee td/osd-backfill-recovery-log/result.log 2020-03-11T22:54:04.827 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=4 2020-03-11T22:54:04.827 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-03-11T22:54:04.827 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/osd-backfill-recovery-log/4 2020-03-11T22:54:04.827 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-backfill-recovery-log/4 --no-mon-config --pgid 1.0 --op log 2020-03-11T22:54:05.454 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2020-03-11T22:54:05.454 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_log_t": { 2020-03-11T22:54:05.454 INFO:tasks.workunit.client.0.smithi059.stdout: "head": "34'150", 2020-03-11T22:54:05.454 INFO:tasks.workunit.client.0.smithi059.stdout: "tail": "34'148", 2020-03-11T22:54:05.454 INFO:tasks.workunit.client.0.smithi059.stdout: "log": [ 2020-03-11T22:54:05.454 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.455 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "modify", 2020-03-11T22:54:05.455 INFO:tasks.workunit.client.0.smithi059.stdout: "object": "1:e9c179f9:::obj-149:head", 2020-03-11T22:54:05.455 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'149", 2020-03-11T22:54:05.455 INFO:tasks.workunit.client.0.smithi059.stdout: "prior_version": "0'0", 2020-03-11T22:54:05.455 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4631.0:1", 2020-03-11T22:54:05.455 INFO:tasks.workunit.client.0.smithi059.stdout: "extra_reqids": [], 2020-03-11T22:54:05.455 INFO:tasks.workunit.client.0.smithi059.stdout: "mtime": "2020-03-11T22:53:30.242554+0000", 2020-03-11T22:54:05.456 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": 0, 2020-03-11T22:54:05.456 INFO:tasks.workunit.client.0.smithi059.stdout: "mod_desc": { 2020-03-11T22:54:05.456 INFO:tasks.workunit.client.0.smithi059.stdout: "object_mod_desc": { 2020-03-11T22:54:05.456 INFO:tasks.workunit.client.0.smithi059.stdout: "can_local_rollback": false, 2020-03-11T22:54:05.456 INFO:tasks.workunit.client.0.smithi059.stdout: "rollback_info_completed": false, 2020-03-11T22:54:05.456 INFO:tasks.workunit.client.0.smithi059.stdout: "ops": [] 2020-03-11T22:54:05.456 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:54:05.457 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.457 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_regions": { 2020-03-11T22:54:05.457 INFO:tasks.workunit.client.0.smithi059.stdout: "object_clean_regions": { 2020-03-11T22:54:05.457 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_offsets": "[6900~18446744073709544715]", 2020-03-11T22:54:05.457 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_omap": true, 2020-03-11T22:54:05.457 INFO:tasks.workunit.client.0.smithi059.stdout: "new_object": false 2020-03-11T22:54:05.457 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:54:05.457 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:54:05.458 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.458 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.458 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "modify", 2020-03-11T22:54:05.458 INFO:tasks.workunit.client.0.smithi059.stdout: "object": "1:cb39fd60:::obj-150:head", 2020-03-11T22:54:05.458 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'150", 2020-03-11T22:54:05.458 INFO:tasks.workunit.client.0.smithi059.stdout: "prior_version": "0'0", 2020-03-11T22:54:05.458 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4634.0:1", 2020-03-11T22:54:05.459 INFO:tasks.workunit.client.0.smithi059.stdout: "extra_reqids": [], 2020-03-11T22:54:05.459 INFO:tasks.workunit.client.0.smithi059.stdout: "mtime": "2020-03-11T22:53:30.376866+0000", 2020-03-11T22:54:05.459 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": 0, 2020-03-11T22:54:05.459 INFO:tasks.workunit.client.0.smithi059.stdout: "mod_desc": { 2020-03-11T22:54:05.459 INFO:tasks.workunit.client.0.smithi059.stdout: "object_mod_desc": { 2020-03-11T22:54:05.459 INFO:tasks.workunit.client.0.smithi059.stdout: "can_local_rollback": false, 2020-03-11T22:54:05.459 INFO:tasks.workunit.client.0.smithi059.stdout: "rollback_info_completed": false, 2020-03-11T22:54:05.459 INFO:tasks.workunit.client.0.smithi059.stdout: "ops": [] 2020-03-11T22:54:05.460 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:54:05.460 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.460 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_regions": { 2020-03-11T22:54:05.460 INFO:tasks.workunit.client.0.smithi059.stdout: "object_clean_regions": { 2020-03-11T22:54:05.460 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_offsets": "[6900~18446744073709544715]", 2020-03-11T22:54:05.460 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_omap": true, 2020-03-11T22:54:05.460 INFO:tasks.workunit.client.0.smithi059.stdout: "new_object": false 2020-03-11T22:54:05.461 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:54:05.461 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:54:05.461 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:54:05.461 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:54:05.461 INFO:tasks.workunit.client.0.smithi059.stdout: "dups": [ 2020-03-11T22:54:05.461 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.461 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4187.0:1", 2020-03-11T22:54:05.462 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'1", 2020-03-11T22:54:05.462 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "1", 2020-03-11T22:54:05.462 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.462 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.462 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.462 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4190.0:1", 2020-03-11T22:54:05.463 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'2", 2020-03-11T22:54:05.463 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "2", 2020-03-11T22:54:05.463 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.463 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.463 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.463 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4193.0:1", 2020-03-11T22:54:05.463 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'3", 2020-03-11T22:54:05.463 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "3", 2020-03-11T22:54:05.464 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.464 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.464 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.464 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4196.0:1", 2020-03-11T22:54:05.464 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'4", 2020-03-11T22:54:05.464 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "4", 2020-03-11T22:54:05.464 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.465 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.465 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.465 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4199.0:1", 2020-03-11T22:54:05.465 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'5", 2020-03-11T22:54:05.465 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "5", 2020-03-11T22:54:05.465 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.465 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.466 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.466 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4202.0:1", 2020-03-11T22:54:05.466 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'6", 2020-03-11T22:54:05.466 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "6", 2020-03-11T22:54:05.466 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.466 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.466 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.466 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4205.0:1", 2020-03-11T22:54:05.467 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'7", 2020-03-11T22:54:05.467 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "7", 2020-03-11T22:54:05.467 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.467 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.467 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.467 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4208.0:1", 2020-03-11T22:54:05.468 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'8", 2020-03-11T22:54:05.468 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "8", 2020-03-11T22:54:05.468 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.468 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.468 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.468 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4211.0:1", 2020-03-11T22:54:05.468 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'9", 2020-03-11T22:54:05.468 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "9", 2020-03-11T22:54:05.469 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.474 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.474 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.474 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4214.0:1", 2020-03-11T22:54:05.474 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'10", 2020-03-11T22:54:05.474 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "10", 2020-03-11T22:54:05.475 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.475 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.475 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.475 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4217.0:1", 2020-03-11T22:54:05.475 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'11", 2020-03-11T22:54:05.475 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "11", 2020-03-11T22:54:05.476 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.476 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.476 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.476 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4220.0:1", 2020-03-11T22:54:05.476 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'12", 2020-03-11T22:54:05.477 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "12", 2020-03-11T22:54:05.477 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.477 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.477 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.477 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4223.0:1", 2020-03-11T22:54:05.477 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'13", 2020-03-11T22:54:05.478 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "13", 2020-03-11T22:54:05.478 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.478 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.478 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.478 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4226.0:1", 2020-03-11T22:54:05.478 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'14", 2020-03-11T22:54:05.478 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "14", 2020-03-11T22:54:05.479 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.479 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.479 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.479 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4229.0:1", 2020-03-11T22:54:05.479 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'15", 2020-03-11T22:54:05.479 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "15", 2020-03-11T22:54:05.479 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.480 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.480 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.480 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4232.0:1", 2020-03-11T22:54:05.480 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'16", 2020-03-11T22:54:05.480 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "16", 2020-03-11T22:54:05.480 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.480 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.481 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.481 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4235.0:1", 2020-03-11T22:54:05.481 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'17", 2020-03-11T22:54:05.481 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "17", 2020-03-11T22:54:05.481 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.481 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.481 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.481 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4238.0:1", 2020-03-11T22:54:05.482 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'18", 2020-03-11T22:54:05.482 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "18", 2020-03-11T22:54:05.482 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.482 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.482 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.482 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4241.0:1", 2020-03-11T22:54:05.482 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'19", 2020-03-11T22:54:05.483 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "19", 2020-03-11T22:54:05.483 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.483 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.483 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.483 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4244.0:1", 2020-03-11T22:54:05.483 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'20", 2020-03-11T22:54:05.483 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "20", 2020-03-11T22:54:05.484 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.484 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.484 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.484 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4247.0:1", 2020-03-11T22:54:05.484 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'21", 2020-03-11T22:54:05.484 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "21", 2020-03-11T22:54:05.484 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.484 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.485 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.485 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4250.0:1", 2020-03-11T22:54:05.485 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'22", 2020-03-11T22:54:05.485 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "22", 2020-03-11T22:54:05.485 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.485 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.485 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.486 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4253.0:1", 2020-03-11T22:54:05.486 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'23", 2020-03-11T22:54:05.486 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "23", 2020-03-11T22:54:05.486 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.486 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.486 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.486 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4256.0:1", 2020-03-11T22:54:05.486 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'24", 2020-03-11T22:54:05.487 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "24", 2020-03-11T22:54:05.487 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.487 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.487 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.487 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4259.0:1", 2020-03-11T22:54:05.487 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'25", 2020-03-11T22:54:05.488 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "25", 2020-03-11T22:54:05.488 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.488 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.488 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.488 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4262.0:1", 2020-03-11T22:54:05.488 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'26", 2020-03-11T22:54:05.488 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "26", 2020-03-11T22:54:05.489 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.489 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.489 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.489 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4265.0:1", 2020-03-11T22:54:05.489 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'27", 2020-03-11T22:54:05.489 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "27", 2020-03-11T22:54:05.489 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.490 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.490 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.490 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4268.0:1", 2020-03-11T22:54:05.490 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'28", 2020-03-11T22:54:05.490 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "28", 2020-03-11T22:54:05.490 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.490 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.490 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.491 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4271.0:1", 2020-03-11T22:54:05.491 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'29", 2020-03-11T22:54:05.491 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "29", 2020-03-11T22:54:05.491 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.491 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.491 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.491 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4274.0:1", 2020-03-11T22:54:05.492 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'30", 2020-03-11T22:54:05.492 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "30", 2020-03-11T22:54:05.492 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.492 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.492 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.492 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4277.0:1", 2020-03-11T22:54:05.492 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'31", 2020-03-11T22:54:05.492 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "31", 2020-03-11T22:54:05.493 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.493 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.493 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.493 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4280.0:1", 2020-03-11T22:54:05.493 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'32", 2020-03-11T22:54:05.493 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "32", 2020-03-11T22:54:05.493 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.494 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.494 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.494 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4283.0:1", 2020-03-11T22:54:05.494 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'33", 2020-03-11T22:54:05.494 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "33", 2020-03-11T22:54:05.494 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.494 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.495 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.495 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4286.0:1", 2020-03-11T22:54:05.495 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'34", 2020-03-11T22:54:05.495 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "34", 2020-03-11T22:54:05.495 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.495 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.496 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.496 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4289.0:1", 2020-03-11T22:54:05.496 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'35", 2020-03-11T22:54:05.496 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "35", 2020-03-11T22:54:05.496 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.496 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.496 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.497 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4292.0:1", 2020-03-11T22:54:05.497 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'36", 2020-03-11T22:54:05.497 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "36", 2020-03-11T22:54:05.497 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.497 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.497 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.497 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4295.0:1", 2020-03-11T22:54:05.498 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'37", 2020-03-11T22:54:05.498 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "37", 2020-03-11T22:54:05.498 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.498 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.498 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.498 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4298.0:1", 2020-03-11T22:54:05.498 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'38", 2020-03-11T22:54:05.499 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "38", 2020-03-11T22:54:05.499 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.499 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.499 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.499 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4301.0:1", 2020-03-11T22:54:05.499 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'39", 2020-03-11T22:54:05.499 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "39", 2020-03-11T22:54:05.499 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.500 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.500 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.500 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4304.0:1", 2020-03-11T22:54:05.500 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'40", 2020-03-11T22:54:05.500 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "40", 2020-03-11T22:54:05.500 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.500 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.501 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.501 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4307.0:1", 2020-03-11T22:54:05.501 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'41", 2020-03-11T22:54:05.501 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "41", 2020-03-11T22:54:05.501 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.501 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.501 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.501 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4310.0:1", 2020-03-11T22:54:05.502 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'42", 2020-03-11T22:54:05.502 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "42", 2020-03-11T22:54:05.502 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.502 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.502 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.502 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4313.0:1", 2020-03-11T22:54:05.502 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'43", 2020-03-11T22:54:05.503 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "43", 2020-03-11T22:54:05.503 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.503 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.503 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.503 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4316.0:1", 2020-03-11T22:54:05.503 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'44", 2020-03-11T22:54:05.503 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "44", 2020-03-11T22:54:05.504 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.504 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.504 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.504 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4319.0:1", 2020-03-11T22:54:05.504 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'45", 2020-03-11T22:54:05.504 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "45", 2020-03-11T22:54:05.504 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.505 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.505 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.505 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4322.0:1", 2020-03-11T22:54:05.505 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'46", 2020-03-11T22:54:05.505 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "46", 2020-03-11T22:54:05.505 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.505 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.505 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.506 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4325.0:1", 2020-03-11T22:54:05.506 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'47", 2020-03-11T22:54:05.506 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "47", 2020-03-11T22:54:05.506 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.506 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.506 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.506 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4328.0:1", 2020-03-11T22:54:05.507 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'48", 2020-03-11T22:54:05.507 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "48", 2020-03-11T22:54:05.507 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.507 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.507 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.507 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4331.0:1", 2020-03-11T22:54:05.507 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'49", 2020-03-11T22:54:05.507 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "49", 2020-03-11T22:54:05.508 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.508 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.508 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.508 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4334.0:1", 2020-03-11T22:54:05.508 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'50", 2020-03-11T22:54:05.508 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "50", 2020-03-11T22:54:05.508 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.509 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.509 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.509 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4337.0:1", 2020-03-11T22:54:05.509 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'51", 2020-03-11T22:54:05.509 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "51", 2020-03-11T22:54:05.509 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.509 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.509 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.510 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4340.0:1", 2020-03-11T22:54:05.510 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'52", 2020-03-11T22:54:05.510 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "52", 2020-03-11T22:54:05.510 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.510 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.510 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.510 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4343.0:1", 2020-03-11T22:54:05.511 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'53", 2020-03-11T22:54:05.511 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "53", 2020-03-11T22:54:05.511 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.511 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.511 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.511 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4346.0:1", 2020-03-11T22:54:05.511 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'54", 2020-03-11T22:54:05.512 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "54", 2020-03-11T22:54:05.512 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.512 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.512 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.512 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4349.0:1", 2020-03-11T22:54:05.512 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'55", 2020-03-11T22:54:05.512 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "55", 2020-03-11T22:54:05.512 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.513 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.513 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.513 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4352.0:1", 2020-03-11T22:54:05.513 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'56", 2020-03-11T22:54:05.513 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "56", 2020-03-11T22:54:05.513 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.513 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.514 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.514 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4355.0:1", 2020-03-11T22:54:05.514 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'57", 2020-03-11T22:54:05.514 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "57", 2020-03-11T22:54:05.514 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.514 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.514 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.514 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4358.0:1", 2020-03-11T22:54:05.515 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'58", 2020-03-11T22:54:05.515 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "58", 2020-03-11T22:54:05.515 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.515 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.515 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.515 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4361.0:1", 2020-03-11T22:54:05.515 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'59", 2020-03-11T22:54:05.516 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "59", 2020-03-11T22:54:05.516 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.516 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.516 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.516 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4364.0:1", 2020-03-11T22:54:05.516 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'60", 2020-03-11T22:54:05.517 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "60", 2020-03-11T22:54:05.517 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.517 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.517 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.517 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4367.0:1", 2020-03-11T22:54:05.517 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'61", 2020-03-11T22:54:05.517 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "61", 2020-03-11T22:54:05.518 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.518 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.518 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.518 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4370.0:1", 2020-03-11T22:54:05.518 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'62", 2020-03-11T22:54:05.518 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "62", 2020-03-11T22:54:05.518 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.519 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.519 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.519 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4373.0:1", 2020-03-11T22:54:05.519 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'63", 2020-03-11T22:54:05.519 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "63", 2020-03-11T22:54:05.519 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.519 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.519 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.520 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4376.0:1", 2020-03-11T22:54:05.520 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'64", 2020-03-11T22:54:05.520 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "64", 2020-03-11T22:54:05.520 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.520 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.520 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.520 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4379.0:1", 2020-03-11T22:54:05.521 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'65", 2020-03-11T22:54:05.521 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "65", 2020-03-11T22:54:05.521 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.521 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.521 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.521 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4382.0:1", 2020-03-11T22:54:05.521 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'66", 2020-03-11T22:54:05.522 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "66", 2020-03-11T22:54:05.522 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.522 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.522 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.522 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4385.0:1", 2020-03-11T22:54:05.522 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'67", 2020-03-11T22:54:05.522 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "67", 2020-03-11T22:54:05.522 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.523 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.523 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.523 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4388.0:1", 2020-03-11T22:54:05.523 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'68", 2020-03-11T22:54:05.523 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "68", 2020-03-11T22:54:05.523 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.523 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.524 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.524 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4391.0:1", 2020-03-11T22:54:05.524 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'69", 2020-03-11T22:54:05.524 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "69", 2020-03-11T22:54:05.524 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.524 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.524 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.524 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4394.0:1", 2020-03-11T22:54:05.525 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'70", 2020-03-11T22:54:05.525 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "70", 2020-03-11T22:54:05.525 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.525 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.525 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.525 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4397.0:1", 2020-03-11T22:54:05.525 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'71", 2020-03-11T22:54:05.526 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "71", 2020-03-11T22:54:05.526 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.526 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.526 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.526 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4400.0:1", 2020-03-11T22:54:05.526 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'72", 2020-03-11T22:54:05.527 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "72", 2020-03-11T22:54:05.527 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.527 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.527 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.527 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4403.0:1", 2020-03-11T22:54:05.527 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'73", 2020-03-11T22:54:05.527 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "73", 2020-03-11T22:54:05.527 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.528 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.528 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.528 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4406.0:1", 2020-03-11T22:54:05.528 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'74", 2020-03-11T22:54:05.528 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "74", 2020-03-11T22:54:05.528 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.528 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.529 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.529 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4409.0:1", 2020-03-11T22:54:05.529 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'75", 2020-03-11T22:54:05.529 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "75", 2020-03-11T22:54:05.529 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.529 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.529 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.530 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4412.0:1", 2020-03-11T22:54:05.530 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'76", 2020-03-11T22:54:05.530 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "76", 2020-03-11T22:54:05.530 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.530 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.530 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.530 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4415.0:1", 2020-03-11T22:54:05.530 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'77", 2020-03-11T22:54:05.531 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "77", 2020-03-11T22:54:05.531 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.531 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.531 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.531 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4418.0:1", 2020-03-11T22:54:05.531 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'78", 2020-03-11T22:54:05.531 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "78", 2020-03-11T22:54:05.532 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.532 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.532 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.532 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4421.0:1", 2020-03-11T22:54:05.532 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'79", 2020-03-11T22:54:05.532 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "79", 2020-03-11T22:54:05.532 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.532 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.533 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.533 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4424.0:1", 2020-03-11T22:54:05.533 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'80", 2020-03-11T22:54:05.533 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "80", 2020-03-11T22:54:05.533 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.533 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.533 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.534 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4427.0:1", 2020-03-11T22:54:05.534 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'81", 2020-03-11T22:54:05.534 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "81", 2020-03-11T22:54:05.534 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.534 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.534 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.534 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4430.0:1", 2020-03-11T22:54:05.535 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'82", 2020-03-11T22:54:05.535 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "82", 2020-03-11T22:54:05.535 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.535 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.535 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.535 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4433.0:1", 2020-03-11T22:54:05.535 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'83", 2020-03-11T22:54:05.535 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "83", 2020-03-11T22:54:05.536 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.536 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.536 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.536 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4436.0:1", 2020-03-11T22:54:05.536 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'84", 2020-03-11T22:54:05.536 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "84", 2020-03-11T22:54:05.536 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.537 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.537 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.537 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4439.0:1", 2020-03-11T22:54:05.537 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'85", 2020-03-11T22:54:05.537 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "85", 2020-03-11T22:54:05.537 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.537 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.538 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.538 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4442.0:1", 2020-03-11T22:54:05.538 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'86", 2020-03-11T22:54:05.538 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "86", 2020-03-11T22:54:05.538 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.538 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.538 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.538 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4445.0:1", 2020-03-11T22:54:05.539 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'87", 2020-03-11T22:54:05.539 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "87", 2020-03-11T22:54:05.539 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.539 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.539 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.539 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4448.0:1", 2020-03-11T22:54:05.539 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'88", 2020-03-11T22:54:05.540 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "88", 2020-03-11T22:54:05.540 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.540 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.540 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.540 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4451.0:1", 2020-03-11T22:54:05.540 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'89", 2020-03-11T22:54:05.540 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "89", 2020-03-11T22:54:05.540 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.541 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.541 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.541 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4454.0:1", 2020-03-11T22:54:05.541 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'90", 2020-03-11T22:54:05.541 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "90", 2020-03-11T22:54:05.541 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.541 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.542 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.542 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4457.0:1", 2020-03-11T22:54:05.542 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'91", 2020-03-11T22:54:05.542 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "91", 2020-03-11T22:54:05.542 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.542 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.542 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.542 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4460.0:1", 2020-03-11T22:54:05.543 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'92", 2020-03-11T22:54:05.543 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "92", 2020-03-11T22:54:05.543 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.543 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.543 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.543 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4463.0:1", 2020-03-11T22:54:05.543 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'93", 2020-03-11T22:54:05.544 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "93", 2020-03-11T22:54:05.544 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.544 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.544 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.544 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4466.0:1", 2020-03-11T22:54:05.544 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'94", 2020-03-11T22:54:05.544 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "94", 2020-03-11T22:54:05.545 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.545 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.545 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.545 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4469.0:1", 2020-03-11T22:54:05.545 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'95", 2020-03-11T22:54:05.545 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "95", 2020-03-11T22:54:05.545 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.546 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.546 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.546 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4472.0:1", 2020-03-11T22:54:05.546 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'96", 2020-03-11T22:54:05.546 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "96", 2020-03-11T22:54:05.546 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.546 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.547 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.547 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4475.0:1", 2020-03-11T22:54:05.547 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'97", 2020-03-11T22:54:05.547 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "97", 2020-03-11T22:54:05.547 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.547 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.547 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.547 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4478.0:1", 2020-03-11T22:54:05.548 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'98", 2020-03-11T22:54:05.548 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "98", 2020-03-11T22:54:05.548 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.548 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.548 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.548 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4481.0:1", 2020-03-11T22:54:05.548 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'99", 2020-03-11T22:54:05.548 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "99", 2020-03-11T22:54:05.549 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.549 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.549 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.549 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4484.0:1", 2020-03-11T22:54:05.549 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'100", 2020-03-11T22:54:05.549 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "100", 2020-03-11T22:54:05.550 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.550 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.550 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.550 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4487.0:1", 2020-03-11T22:54:05.550 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'101", 2020-03-11T22:54:05.550 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "101", 2020-03-11T22:54:05.550 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.551 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.551 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.551 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4490.0:1", 2020-03-11T22:54:05.551 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'102", 2020-03-11T22:54:05.551 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "102", 2020-03-11T22:54:05.551 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.551 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.552 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.552 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4493.0:1", 2020-03-11T22:54:05.552 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'103", 2020-03-11T22:54:05.552 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "103", 2020-03-11T22:54:05.552 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.552 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.552 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.552 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4496.0:1", 2020-03-11T22:54:05.553 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'104", 2020-03-11T22:54:05.553 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "104", 2020-03-11T22:54:05.553 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.553 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.553 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.553 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4499.0:1", 2020-03-11T22:54:05.553 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'105", 2020-03-11T22:54:05.554 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "105", 2020-03-11T22:54:05.554 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.554 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.554 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.554 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4502.0:1", 2020-03-11T22:54:05.554 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'106", 2020-03-11T22:54:05.555 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "106", 2020-03-11T22:54:05.555 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.555 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.555 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.555 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4505.0:1", 2020-03-11T22:54:05.555 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'107", 2020-03-11T22:54:05.555 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "107", 2020-03-11T22:54:05.556 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.556 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.556 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.556 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4508.0:1", 2020-03-11T22:54:05.556 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'108", 2020-03-11T22:54:05.556 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "108", 2020-03-11T22:54:05.556 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.557 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.557 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.557 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4511.0:1", 2020-03-11T22:54:05.557 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'109", 2020-03-11T22:54:05.557 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "109", 2020-03-11T22:54:05.557 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.557 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.558 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.558 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4514.0:1", 2020-03-11T22:54:05.558 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'110", 2020-03-11T22:54:05.558 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "110", 2020-03-11T22:54:05.558 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.558 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.558 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.559 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4517.0:1", 2020-03-11T22:54:05.559 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'111", 2020-03-11T22:54:05.559 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "111", 2020-03-11T22:54:05.559 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.559 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.559 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.559 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4520.0:1", 2020-03-11T22:54:05.559 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'112", 2020-03-11T22:54:05.560 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "112", 2020-03-11T22:54:05.560 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.560 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.560 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.560 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4523.0:1", 2020-03-11T22:54:05.560 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'113", 2020-03-11T22:54:05.560 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "113", 2020-03-11T22:54:05.561 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.561 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.561 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.561 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4526.0:1", 2020-03-11T22:54:05.561 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'114", 2020-03-11T22:54:05.561 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "114", 2020-03-11T22:54:05.561 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.562 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.562 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.562 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4529.0:1", 2020-03-11T22:54:05.562 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'115", 2020-03-11T22:54:05.562 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "115", 2020-03-11T22:54:05.562 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.562 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.563 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.563 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4532.0:1", 2020-03-11T22:54:05.563 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'116", 2020-03-11T22:54:05.563 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "116", 2020-03-11T22:54:05.563 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.563 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.563 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.563 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4535.0:1", 2020-03-11T22:54:05.564 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'117", 2020-03-11T22:54:05.564 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "117", 2020-03-11T22:54:05.564 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.564 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.564 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.564 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4538.0:1", 2020-03-11T22:54:05.564 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'118", 2020-03-11T22:54:05.565 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "118", 2020-03-11T22:54:05.565 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.565 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.565 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.565 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4541.0:1", 2020-03-11T22:54:05.565 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'119", 2020-03-11T22:54:05.565 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "119", 2020-03-11T22:54:05.566 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.566 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.566 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.566 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4544.0:1", 2020-03-11T22:54:05.566 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'120", 2020-03-11T22:54:05.566 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "120", 2020-03-11T22:54:05.566 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.567 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.567 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.567 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4547.0:1", 2020-03-11T22:54:05.567 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'121", 2020-03-11T22:54:05.567 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "121", 2020-03-11T22:54:05.567 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.567 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.568 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.568 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4550.0:1", 2020-03-11T22:54:05.568 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'122", 2020-03-11T22:54:05.568 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "122", 2020-03-11T22:54:05.568 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.568 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.568 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.568 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4553.0:1", 2020-03-11T22:54:05.569 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'123", 2020-03-11T22:54:05.569 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "123", 2020-03-11T22:54:05.569 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.569 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.569 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.569 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4556.0:1", 2020-03-11T22:54:05.569 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'124", 2020-03-11T22:54:05.570 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "124", 2020-03-11T22:54:05.570 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.570 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.570 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.570 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4559.0:1", 2020-03-11T22:54:05.570 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'125", 2020-03-11T22:54:05.570 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "125", 2020-03-11T22:54:05.571 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.571 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.571 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.571 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4562.0:1", 2020-03-11T22:54:05.571 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'126", 2020-03-11T22:54:05.571 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "126", 2020-03-11T22:54:05.571 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.572 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.572 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.572 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4565.0:1", 2020-03-11T22:54:05.572 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'127", 2020-03-11T22:54:05.572 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "127", 2020-03-11T22:54:05.572 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.572 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.573 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.573 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4568.0:1", 2020-03-11T22:54:05.573 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'128", 2020-03-11T22:54:05.573 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "128", 2020-03-11T22:54:05.573 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.573 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.573 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.573 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4571.0:1", 2020-03-11T22:54:05.574 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'129", 2020-03-11T22:54:05.574 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "129", 2020-03-11T22:54:05.574 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.574 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.574 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.574 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4574.0:1", 2020-03-11T22:54:05.574 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'130", 2020-03-11T22:54:05.575 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "130", 2020-03-11T22:54:05.575 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.575 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.575 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.575 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4577.0:1", 2020-03-11T22:54:05.575 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'131", 2020-03-11T22:54:05.575 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "131", 2020-03-11T22:54:05.575 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.576 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.576 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.576 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4580.0:1", 2020-03-11T22:54:05.576 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'132", 2020-03-11T22:54:05.576 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "132", 2020-03-11T22:54:05.576 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.577 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.577 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.577 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4583.0:1", 2020-03-11T22:54:05.577 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'133", 2020-03-11T22:54:05.578 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "133", 2020-03-11T22:54:05.578 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.578 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.578 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.578 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4586.0:1", 2020-03-11T22:54:05.578 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'134", 2020-03-11T22:54:05.578 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "134", 2020-03-11T22:54:05.579 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.579 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.579 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.579 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4589.0:1", 2020-03-11T22:54:05.579 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'135", 2020-03-11T22:54:05.579 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "135", 2020-03-11T22:54:05.579 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.579 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.580 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.580 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4592.0:1", 2020-03-11T22:54:05.580 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'136", 2020-03-11T22:54:05.580 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "136", 2020-03-11T22:54:05.580 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.580 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.580 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.581 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4595.0:1", 2020-03-11T22:54:05.581 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'137", 2020-03-11T22:54:05.581 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "137", 2020-03-11T22:54:05.581 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.581 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.581 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.581 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4598.0:1", 2020-03-11T22:54:05.581 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'138", 2020-03-11T22:54:05.582 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "138", 2020-03-11T22:54:05.582 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.582 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.582 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.582 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4601.0:1", 2020-03-11T22:54:05.582 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'139", 2020-03-11T22:54:05.582 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "139", 2020-03-11T22:54:05.583 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.583 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.583 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.583 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4604.0:1", 2020-03-11T22:54:05.583 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'140", 2020-03-11T22:54:05.583 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "140", 2020-03-11T22:54:05.583 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.583 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.584 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.584 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4607.0:1", 2020-03-11T22:54:05.584 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'141", 2020-03-11T22:54:05.584 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "141", 2020-03-11T22:54:05.584 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.584 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.584 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.585 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4610.0:1", 2020-03-11T22:54:05.585 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'142", 2020-03-11T22:54:05.585 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "142", 2020-03-11T22:54:05.585 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.585 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.585 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.585 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4613.0:1", 2020-03-11T22:54:05.586 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'143", 2020-03-11T22:54:05.586 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "143", 2020-03-11T22:54:05.586 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.586 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.586 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.586 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4616.0:1", 2020-03-11T22:54:05.586 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'144", 2020-03-11T22:54:05.586 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "144", 2020-03-11T22:54:05.587 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.587 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.587 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.587 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4619.0:1", 2020-03-11T22:54:05.587 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'145", 2020-03-11T22:54:05.587 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "145", 2020-03-11T22:54:05.587 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.588 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.588 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.588 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4622.0:1", 2020-03-11T22:54:05.588 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'146", 2020-03-11T22:54:05.588 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "146", 2020-03-11T22:54:05.588 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.589 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.589 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.589 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4625.0:1", 2020-03-11T22:54:05.589 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'147", 2020-03-11T22:54:05.966 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "147", 2020-03-11T22:54:05.967 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.967 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.967 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:54:05.967 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4628.0:1", 2020-03-11T22:54:05.967 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'148", 2020-03-11T22:54:05.967 INFO:tasks.workunit.client.0.smithi059.stdout: "user_version": "148", 2020-03-11T22:54:05.967 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": "0" 2020-03-11T22:54:05.968 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:54:05.968 INFO:tasks.workunit.client.0.smithi059.stdout: ] 2020-03-11T22:54:05.968 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:54:05.968 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_missing_t": { 2020-03-11T22:54:05.968 INFO:tasks.workunit.client.0.smithi059.stdout: "missing": [], 2020-03-11T22:54:05.968 INFO:tasks.workunit.client.0.smithi059.stdout: "may_include_deletes": true 2020-03-11T22:54:05.968 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:54:05.968 INFO:tasks.workunit.client.0.smithi059.stdout:} 2020-03-11T22:54:05.969 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:82: _common_test: jq '.pg_log_t.log | length' td/osd-backfill-recovery-log/result.log 2020-03-11T22:54:05.971 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:82: _common_test: LOGLEN=2 2020-03-11T22:54:05.971 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:83: _common_test: '[' 2 '!=' 2 ']' 2020-03-11T22:54:05.972 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:87: _common_test: jq '.pg_log_t.dups | length' td/osd-backfill-recovery-log/result.log 2020-03-11T22:54:05.974 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:87: _common_test: DUPSLEN=148 2020-03-11T22:54:05.975 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:88: _common_test: '[' 148 '!=' 148 ']' 2020-03-11T22:54:05.975 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:92: _common_test: grep 'copy_up_to\|copy_after' td/osd-backfill-recovery-log/osd.0.log td/osd-backfill-recovery-log/osd.1.log td/osd-backfill-recovery-log/osd.2.log td/osd-backfill-recovery-log/osd.3.log td/osd-backfill-recovery-log/osd.4.log td/osd-backfill-recovery-log/osd.5.log 2020-03-11T22:54:05.977 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to max 2 2020-03-11T22:54:05.978 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to copy log version 34'150 2020-03-11T22:54:05.978 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to copy log version 34'149 2020-03-11T22:54:05.978 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after earliest_dup_version 0 2020-03-11T22:54:05.978 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'1 2020-03-11T22:54:05.978 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'2 2020-03-11T22:54:05.978 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'3 2020-03-11T22:54:05.979 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'4 2020-03-11T22:54:05.979 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'5 2020-03-11T22:54:05.979 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'6 2020-03-11T22:54:05.979 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'7 2020-03-11T22:54:05.979 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'8 2020-03-11T22:54:05.979 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'9 2020-03-11T22:54:05.979 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'10 2020-03-11T22:54:05.980 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'11 2020-03-11T22:54:05.980 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'12 2020-03-11T22:54:05.980 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'13 2020-03-11T22:54:05.980 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'14 2020-03-11T22:54:05.980 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'15 2020-03-11T22:54:05.981 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'16 2020-03-11T22:54:05.981 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'17 2020-03-11T22:54:05.981 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'18 2020-03-11T22:54:05.981 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'19 2020-03-11T22:54:05.981 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'20 2020-03-11T22:54:05.981 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'21 2020-03-11T22:54:05.981 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'22 2020-03-11T22:54:05.982 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'23 2020-03-11T22:54:05.982 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'24 2020-03-11T22:54:05.982 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'25 2020-03-11T22:54:05.982 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'26 2020-03-11T22:54:05.982 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'27 2020-03-11T22:54:05.982 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'28 2020-03-11T22:54:05.982 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'29 2020-03-11T22:54:05.983 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'30 2020-03-11T22:54:05.983 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'31 2020-03-11T22:54:05.983 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'32 2020-03-11T22:54:05.983 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'33 2020-03-11T22:54:05.983 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'34 2020-03-11T22:54:05.983 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'35 2020-03-11T22:54:05.983 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'36 2020-03-11T22:54:05.983 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'37 2020-03-11T22:54:05.984 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'38 2020-03-11T22:54:05.984 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'39 2020-03-11T22:54:05.984 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'40 2020-03-11T22:54:05.984 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'41 2020-03-11T22:54:05.984 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'42 2020-03-11T22:54:05.984 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'43 2020-03-11T22:54:05.984 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'44 2020-03-11T22:54:05.985 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'45 2020-03-11T22:54:05.985 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'46 2020-03-11T22:54:05.985 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'47 2020-03-11T22:54:05.985 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'48 2020-03-11T22:54:05.985 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'49 2020-03-11T22:54:05.985 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'50 2020-03-11T22:54:05.985 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'51 2020-03-11T22:54:05.986 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'52 2020-03-11T22:54:05.986 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'53 2020-03-11T22:54:05.987 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'54 2020-03-11T22:54:05.987 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'55 2020-03-11T22:54:05.987 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'56 2020-03-11T22:54:05.987 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'57 2020-03-11T22:54:05.987 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'58 2020-03-11T22:54:05.987 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'59 2020-03-11T22:54:05.987 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'60 2020-03-11T22:54:05.988 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'61 2020-03-11T22:54:05.988 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'62 2020-03-11T22:54:05.988 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'63 2020-03-11T22:54:05.988 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'64 2020-03-11T22:54:05.988 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'65 2020-03-11T22:54:05.988 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'66 2020-03-11T22:54:05.988 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'67 2020-03-11T22:54:05.988 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'68 2020-03-11T22:54:05.989 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'69 2020-03-11T22:54:05.989 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'70 2020-03-11T22:54:05.989 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'71 2020-03-11T22:54:05.989 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'72 2020-03-11T22:54:05.989 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'73 2020-03-11T22:54:05.989 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'74 2020-03-11T22:54:05.989 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'75 2020-03-11T22:54:05.990 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'76 2020-03-11T22:54:05.990 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'77 2020-03-11T22:54:05.990 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'78 2020-03-11T22:54:05.990 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'79 2020-03-11T22:54:05.990 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'80 2020-03-11T22:54:05.990 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'81 2020-03-11T22:54:05.990 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'82 2020-03-11T22:54:05.991 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'83 2020-03-11T22:54:05.991 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'84 2020-03-11T22:54:05.991 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'85 2020-03-11T22:54:05.991 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'86 2020-03-11T22:54:05.991 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'87 2020-03-11T22:54:05.991 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'88 2020-03-11T22:54:05.992 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'89 2020-03-11T22:54:05.992 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'90 2020-03-11T22:54:05.992 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'91 2020-03-11T22:54:05.992 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'92 2020-03-11T22:54:05.992 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'93 2020-03-11T22:54:05.992 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'94 2020-03-11T22:54:05.992 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'95 2020-03-11T22:54:05.992 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'96 2020-03-11T22:54:05.993 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'97 2020-03-11T22:54:05.993 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'98 2020-03-11T22:54:05.993 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'99 2020-03-11T22:54:05.993 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'100 2020-03-11T22:54:05.993 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'101 2020-03-11T22:54:05.993 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'102 2020-03-11T22:54:05.993 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'103 2020-03-11T22:54:05.994 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'104 2020-03-11T22:54:05.994 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'105 2020-03-11T22:54:05.994 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'106 2020-03-11T22:54:05.994 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'107 2020-03-11T22:54:05.994 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'108 2020-03-11T22:54:05.994 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'109 2020-03-11T22:54:05.994 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'110 2020-03-11T22:54:05.995 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'111 2020-03-11T22:54:05.995 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'112 2020-03-11T22:54:05.995 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'113 2020-03-11T22:54:05.995 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'114 2020-03-11T22:54:05.995 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'115 2020-03-11T22:54:05.995 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'116 2020-03-11T22:54:05.995 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'117 2020-03-11T22:54:05.995 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'118 2020-03-11T22:54:05.996 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'119 2020-03-11T22:54:05.996 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'120 2020-03-11T22:54:05.996 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'121 2020-03-11T22:54:05.996 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'122 2020-03-11T22:54:05.996 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'123 2020-03-11T22:54:05.996 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'124 2020-03-11T22:54:05.997 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'125 2020-03-11T22:54:05.997 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'126 2020-03-11T22:54:05.997 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'127 2020-03-11T22:54:05.997 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'128 2020-03-11T22:54:05.997 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'129 2020-03-11T22:54:05.997 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'130 2020-03-11T22:54:05.998 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'131 2020-03-11T22:54:05.998 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'132 2020-03-11T22:54:05.998 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'133 2020-03-11T22:54:05.999 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'134 2020-03-11T22:54:05.999 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'135 2020-03-11T22:54:05.999 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'136 2020-03-11T22:54:05.999 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'137 2020-03-11T22:54:05.999 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'138 2020-03-11T22:54:06.000 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'139 2020-03-11T22:54:06.000 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'140 2020-03-11T22:54:06.000 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'141 2020-03-11T22:54:06.000 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'142 2020-03-11T22:54:06.001 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'143 2020-03-11T22:54:06.001 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'144 2020-03-11T22:54:06.001 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'145 2020-03-11T22:54:06.001 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'146 2020-03-11T22:54:06.002 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'147 2020-03-11T22:54:06.002 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'148 2020-03-11T22:54:06.002 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to max 2 2020-03-11T22:54:06.002 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to copy log version 34'150 2020-03-11T22:54:06.003 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to copy log version 34'149 2020-03-11T22:54:06.003 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after earliest_dup_version 0 2020-03-11T22:54:06.003 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'1 2020-03-11T22:54:06.003 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'2 2020-03-11T22:54:06.004 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'3 2020-03-11T22:54:06.004 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'4 2020-03-11T22:54:06.004 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'5 2020-03-11T22:54:06.004 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'6 2020-03-11T22:54:06.005 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'7 2020-03-11T22:54:06.005 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'8 2020-03-11T22:54:06.005 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'9 2020-03-11T22:54:06.005 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'10 2020-03-11T22:54:06.005 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'11 2020-03-11T22:54:06.006 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'12 2020-03-11T22:54:06.006 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'13 2020-03-11T22:54:06.006 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'14 2020-03-11T22:54:06.006 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'15 2020-03-11T22:54:06.007 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'16 2020-03-11T22:54:06.007 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'17 2020-03-11T22:54:06.007 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'18 2020-03-11T22:54:06.007 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'19 2020-03-11T22:54:06.008 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'20 2020-03-11T22:54:06.008 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'21 2020-03-11T22:54:06.008 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'22 2020-03-11T22:54:06.008 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'23 2020-03-11T22:54:06.009 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'24 2020-03-11T22:54:06.009 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'25 2020-03-11T22:54:06.009 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'26 2020-03-11T22:54:06.009 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'27 2020-03-11T22:54:06.010 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'28 2020-03-11T22:54:06.010 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'29 2020-03-11T22:54:06.010 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'30 2020-03-11T22:54:06.010 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'31 2020-03-11T22:54:06.010 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'32 2020-03-11T22:54:06.011 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'33 2020-03-11T22:54:06.011 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'34 2020-03-11T22:54:06.011 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'35 2020-03-11T22:54:06.011 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'36 2020-03-11T22:54:06.012 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'37 2020-03-11T22:54:06.012 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'38 2020-03-11T22:54:06.012 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'39 2020-03-11T22:54:06.012 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'40 2020-03-11T22:54:06.013 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'41 2020-03-11T22:54:06.013 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'42 2020-03-11T22:54:06.013 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'43 2020-03-11T22:54:06.013 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'44 2020-03-11T22:54:06.013 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'45 2020-03-11T22:54:06.014 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'46 2020-03-11T22:54:06.014 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'47 2020-03-11T22:54:06.014 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'48 2020-03-11T22:54:06.014 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'49 2020-03-11T22:54:06.015 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'50 2020-03-11T22:54:06.015 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'51 2020-03-11T22:54:06.015 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'52 2020-03-11T22:54:06.015 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'53 2020-03-11T22:54:06.015 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'54 2020-03-11T22:54:06.016 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'55 2020-03-11T22:54:06.016 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'56 2020-03-11T22:54:06.016 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'57 2020-03-11T22:54:06.016 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'58 2020-03-11T22:54:06.017 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'59 2020-03-11T22:54:06.017 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'60 2020-03-11T22:54:06.017 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'61 2020-03-11T22:54:06.017 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'62 2020-03-11T22:54:06.018 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'63 2020-03-11T22:54:06.018 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'64 2020-03-11T22:54:06.018 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'65 2020-03-11T22:54:06.018 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'66 2020-03-11T22:54:06.018 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'67 2020-03-11T22:54:06.019 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'68 2020-03-11T22:54:06.019 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'69 2020-03-11T22:54:06.019 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'70 2020-03-11T22:54:06.019 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'71 2020-03-11T22:54:06.020 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'72 2020-03-11T22:54:06.020 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'73 2020-03-11T22:54:06.020 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'74 2020-03-11T22:54:06.021 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'75 2020-03-11T22:54:06.021 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'76 2020-03-11T22:54:06.021 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'77 2020-03-11T22:54:06.021 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'78 2020-03-11T22:54:06.022 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'79 2020-03-11T22:54:06.022 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'80 2020-03-11T22:54:06.022 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'81 2020-03-11T22:54:06.022 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'82 2020-03-11T22:54:06.022 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'83 2020-03-11T22:54:06.023 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'84 2020-03-11T22:54:06.023 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'85 2020-03-11T22:54:06.023 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'86 2020-03-11T22:54:06.023 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'87 2020-03-11T22:54:06.024 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'88 2020-03-11T22:54:06.024 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'89 2020-03-11T22:54:06.024 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'90 2020-03-11T22:54:06.024 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'91 2020-03-11T22:54:06.025 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'92 2020-03-11T22:54:06.025 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'93 2020-03-11T22:54:06.025 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'94 2020-03-11T22:54:06.025 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'95 2020-03-11T22:54:06.026 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'96 2020-03-11T22:54:06.026 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'97 2020-03-11T22:54:06.026 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'98 2020-03-11T22:54:06.027 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'99 2020-03-11T22:54:06.027 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'100 2020-03-11T22:54:06.027 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'101 2020-03-11T22:54:06.027 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'102 2020-03-11T22:54:06.028 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'103 2020-03-11T22:54:06.028 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'104 2020-03-11T22:54:06.028 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'105 2020-03-11T22:54:06.028 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'106 2020-03-11T22:54:06.029 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'107 2020-03-11T22:54:06.029 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'108 2020-03-11T22:54:06.029 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'109 2020-03-11T22:54:06.029 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'110 2020-03-11T22:54:06.030 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'111 2020-03-11T22:54:06.030 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'112 2020-03-11T22:54:06.030 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'113 2020-03-11T22:54:06.030 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'114 2020-03-11T22:54:06.031 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'115 2020-03-11T22:54:06.031 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'116 2020-03-11T22:54:06.033 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'117 2020-03-11T22:54:06.033 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'118 2020-03-11T22:54:06.033 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'119 2020-03-11T22:54:06.033 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'120 2020-03-11T22:54:06.033 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'121 2020-03-11T22:54:06.034 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'122 2020-03-11T22:54:06.034 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'123 2020-03-11T22:54:06.034 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'124 2020-03-11T22:54:06.034 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'125 2020-03-11T22:54:06.035 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'126 2020-03-11T22:54:06.035 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'127 2020-03-11T22:54:06.035 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'128 2020-03-11T22:54:06.035 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'129 2020-03-11T22:54:06.036 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'130 2020-03-11T22:54:06.036 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'131 2020-03-11T22:54:06.036 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'132 2020-03-11T22:54:06.036 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'133 2020-03-11T22:54:06.036 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'134 2020-03-11T22:54:06.037 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'135 2020-03-11T22:54:06.037 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'136 2020-03-11T22:54:06.037 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'137 2020-03-11T22:54:06.037 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'138 2020-03-11T22:54:06.038 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'139 2020-03-11T22:54:06.038 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'140 2020-03-11T22:54:06.038 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'141 2020-03-11T22:54:06.038 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'142 2020-03-11T22:54:06.038 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'143 2020-03-11T22:54:06.038 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'144 2020-03-11T22:54:06.039 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'145 2020-03-11T22:54:06.039 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'146 2020-03-11T22:54:06.039 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'147 2020-03-11T22:54:06.039 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'148 2020-03-11T22:54:06.039 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to max 2 2020-03-11T22:54:06.039 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to copy log version 34'150 2020-03-11T22:54:06.039 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to copy log version 34'149 2020-03-11T22:54:06.039 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after earliest_dup_version 0 2020-03-11T22:54:06.040 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'1 2020-03-11T22:54:06.040 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'2 2020-03-11T22:54:06.040 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'3 2020-03-11T22:54:06.040 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'4 2020-03-11T22:54:06.040 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'5 2020-03-11T22:54:06.040 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'6 2020-03-11T22:54:06.040 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'7 2020-03-11T22:54:06.041 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'8 2020-03-11T22:54:06.041 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'9 2020-03-11T22:54:06.041 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'10 2020-03-11T22:54:06.041 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'11 2020-03-11T22:54:06.041 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'12 2020-03-11T22:54:06.041 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'13 2020-03-11T22:54:06.041 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'14 2020-03-11T22:54:06.042 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'15 2020-03-11T22:54:06.042 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'16 2020-03-11T22:54:06.042 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'17 2020-03-11T22:54:06.042 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'18 2020-03-11T22:54:06.042 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'19 2020-03-11T22:54:06.042 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'20 2020-03-11T22:54:06.042 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'21 2020-03-11T22:54:06.043 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'22 2020-03-11T22:54:06.043 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'23 2020-03-11T22:54:06.043 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'24 2020-03-11T22:54:06.043 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'25 2020-03-11T22:54:06.043 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'26 2020-03-11T22:54:06.043 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'27 2020-03-11T22:54:06.043 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'28 2020-03-11T22:54:06.043 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'29 2020-03-11T22:54:06.044 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'30 2020-03-11T22:54:06.044 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'31 2020-03-11T22:54:06.044 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'32 2020-03-11T22:54:06.044 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'33 2020-03-11T22:54:06.044 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'34 2020-03-11T22:54:06.044 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'35 2020-03-11T22:54:06.044 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'36 2020-03-11T22:54:06.045 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'37 2020-03-11T22:54:06.045 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'38 2020-03-11T22:54:06.045 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'39 2020-03-11T22:54:06.045 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'40 2020-03-11T22:54:06.045 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'41 2020-03-11T22:54:06.045 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'42 2020-03-11T22:54:06.045 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'43 2020-03-11T22:54:06.046 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'44 2020-03-11T22:54:06.046 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'45 2020-03-11T22:54:06.046 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'46 2020-03-11T22:54:06.046 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'47 2020-03-11T22:54:06.046 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'48 2020-03-11T22:54:06.046 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'49 2020-03-11T22:54:06.046 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'50 2020-03-11T22:54:06.046 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'51 2020-03-11T22:54:06.047 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'52 2020-03-11T22:54:06.047 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'53 2020-03-11T22:54:06.047 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'54 2020-03-11T22:54:06.047 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'55 2020-03-11T22:54:06.047 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'56 2020-03-11T22:54:06.047 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'57 2020-03-11T22:54:06.047 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'58 2020-03-11T22:54:06.048 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'59 2020-03-11T22:54:06.048 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'60 2020-03-11T22:54:06.048 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'61 2020-03-11T22:54:06.048 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'62 2020-03-11T22:54:06.048 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'63 2020-03-11T22:54:06.048 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'64 2020-03-11T22:54:06.048 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'65 2020-03-11T22:54:06.049 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'66 2020-03-11T22:54:06.049 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'67 2020-03-11T22:54:06.049 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'68 2020-03-11T22:54:06.049 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'69 2020-03-11T22:54:06.049 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'70 2020-03-11T22:54:06.049 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'71 2020-03-11T22:54:06.049 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'72 2020-03-11T22:54:06.050 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'73 2020-03-11T22:54:06.050 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'74 2020-03-11T22:54:06.050 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'75 2020-03-11T22:54:06.050 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'76 2020-03-11T22:54:06.050 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'77 2020-03-11T22:54:06.051 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'78 2020-03-11T22:54:06.051 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'79 2020-03-11T22:54:06.051 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'80 2020-03-11T22:54:06.051 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'81 2020-03-11T22:54:06.051 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'82 2020-03-11T22:54:06.051 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'83 2020-03-11T22:54:06.051 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'84 2020-03-11T22:54:06.052 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'85 2020-03-11T22:54:06.052 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'86 2020-03-11T22:54:06.052 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'87 2020-03-11T22:54:06.052 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'88 2020-03-11T22:54:06.052 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'89 2020-03-11T22:54:06.052 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'90 2020-03-11T22:54:06.052 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'91 2020-03-11T22:54:06.052 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'92 2020-03-11T22:54:06.053 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'93 2020-03-11T22:54:06.053 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'94 2020-03-11T22:54:06.053 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'95 2020-03-11T22:54:06.053 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'96 2020-03-11T22:54:06.053 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'97 2020-03-11T22:54:06.053 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'98 2020-03-11T22:54:06.053 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'99 2020-03-11T22:54:06.054 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'100 2020-03-11T22:54:06.054 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'101 2020-03-11T22:54:06.054 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'102 2020-03-11T22:54:06.054 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'103 2020-03-11T22:54:06.054 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'104 2020-03-11T22:54:06.054 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'105 2020-03-11T22:54:06.054 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'106 2020-03-11T22:54:06.055 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'107 2020-03-11T22:54:06.055 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'108 2020-03-11T22:54:06.055 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'109 2020-03-11T22:54:06.055 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'110 2020-03-11T22:54:06.055 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'111 2020-03-11T22:54:06.055 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'112 2020-03-11T22:54:06.055 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'113 2020-03-11T22:54:06.056 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'114 2020-03-11T22:54:06.056 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'115 2020-03-11T22:54:06.056 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'116 2020-03-11T22:54:06.056 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'117 2020-03-11T22:54:06.056 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'118 2020-03-11T22:54:06.056 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'119 2020-03-11T22:54:06.056 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'120 2020-03-11T22:54:06.056 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'121 2020-03-11T22:54:06.057 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'122 2020-03-11T22:54:06.057 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'123 2020-03-11T22:54:06.057 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'124 2020-03-11T22:54:06.057 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'125 2020-03-11T22:54:06.058 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'126 2020-03-11T22:54:06.058 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'127 2020-03-11T22:54:06.058 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'128 2020-03-11T22:54:06.066 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'129 2020-03-11T22:54:06.066 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'130 2020-03-11T22:54:06.066 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'131 2020-03-11T22:54:06.066 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'132 2020-03-11T22:54:06.066 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'133 2020-03-11T22:54:06.067 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'134 2020-03-11T22:54:06.067 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'135 2020-03-11T22:54:06.067 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'136 2020-03-11T22:54:06.067 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'137 2020-03-11T22:54:06.067 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'138 2020-03-11T22:54:06.067 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'139 2020-03-11T22:54:06.069 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'140 2020-03-11T22:54:06.069 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'141 2020-03-11T22:54:06.069 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'142 2020-03-11T22:54:06.070 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'143 2020-03-11T22:54:06.070 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'144 2020-03-11T22:54:06.070 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'145 2020-03-11T22:54:06.070 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'146 2020-03-11T22:54:06.070 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'147 2020-03-11T22:54:06.070 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:53:32.987+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'148 2020-03-11T22:54:06.070 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_after v 34'150 2020-03-11T22:54:06.071 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after earliest_dup_version 0 2020-03-11T22:54:06.071 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'1 2020-03-11T22:54:06.071 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'2 2020-03-11T22:54:06.071 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'3 2020-03-11T22:54:06.071 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'4 2020-03-11T22:54:06.071 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'5 2020-03-11T22:54:06.072 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'6 2020-03-11T22:54:06.072 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'7 2020-03-11T22:54:06.072 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'8 2020-03-11T22:54:06.072 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'9 2020-03-11T22:54:06.072 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'10 2020-03-11T22:54:06.072 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'11 2020-03-11T22:54:06.073 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'12 2020-03-11T22:54:06.073 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'13 2020-03-11T22:54:06.073 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'14 2020-03-11T22:54:06.073 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'15 2020-03-11T22:54:06.073 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'16 2020-03-11T22:54:06.073 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'17 2020-03-11T22:54:06.073 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'18 2020-03-11T22:54:06.074 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'19 2020-03-11T22:54:06.074 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'20 2020-03-11T22:54:06.074 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'21 2020-03-11T22:54:06.074 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'22 2020-03-11T22:54:06.074 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'23 2020-03-11T22:54:06.074 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'24 2020-03-11T22:54:06.075 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'25 2020-03-11T22:54:06.075 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'26 2020-03-11T22:54:06.075 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'27 2020-03-11T22:54:06.075 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'28 2020-03-11T22:54:06.075 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'29 2020-03-11T22:54:06.075 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'30 2020-03-11T22:54:06.076 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'31 2020-03-11T22:54:06.076 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'32 2020-03-11T22:54:06.076 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'33 2020-03-11T22:54:06.076 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'34 2020-03-11T22:54:06.076 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'35 2020-03-11T22:54:06.076 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'36 2020-03-11T22:54:06.077 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'37 2020-03-11T22:54:06.077 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'38 2020-03-11T22:54:06.077 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'39 2020-03-11T22:54:06.077 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'40 2020-03-11T22:54:06.077 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'41 2020-03-11T22:54:06.077 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'42 2020-03-11T22:54:06.077 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'43 2020-03-11T22:54:06.078 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'44 2020-03-11T22:54:06.078 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'45 2020-03-11T22:54:06.078 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'46 2020-03-11T22:54:06.078 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'47 2020-03-11T22:54:06.078 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'48 2020-03-11T22:54:06.078 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'49 2020-03-11T22:54:06.079 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'50 2020-03-11T22:54:06.079 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'51 2020-03-11T22:54:06.079 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'52 2020-03-11T22:54:06.079 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'53 2020-03-11T22:54:06.079 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'54 2020-03-11T22:54:06.080 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'55 2020-03-11T22:54:06.080 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'56 2020-03-11T22:54:06.080 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'57 2020-03-11T22:54:06.080 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'58 2020-03-11T22:54:06.080 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'59 2020-03-11T22:54:06.080 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'60 2020-03-11T22:54:06.080 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'61 2020-03-11T22:54:06.081 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'62 2020-03-11T22:54:06.081 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'63 2020-03-11T22:54:06.081 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'64 2020-03-11T22:54:06.081 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'65 2020-03-11T22:54:06.081 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'66 2020-03-11T22:54:06.081 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'67 2020-03-11T22:54:06.081 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'68 2020-03-11T22:54:06.082 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'69 2020-03-11T22:54:06.082 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'70 2020-03-11T22:54:06.082 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'71 2020-03-11T22:54:06.082 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'72 2020-03-11T22:54:06.082 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'73 2020-03-11T22:54:06.083 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'74 2020-03-11T22:54:06.083 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'75 2020-03-11T22:54:06.083 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'76 2020-03-11T22:54:06.083 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'77 2020-03-11T22:54:06.083 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'78 2020-03-11T22:54:06.084 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'79 2020-03-11T22:54:06.084 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'80 2020-03-11T22:54:06.084 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'81 2020-03-11T22:54:06.084 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'82 2020-03-11T22:54:06.084 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'83 2020-03-11T22:54:06.084 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'84 2020-03-11T22:54:06.084 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'85 2020-03-11T22:54:06.085 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'86 2020-03-11T22:54:06.085 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'87 2020-03-11T22:54:06.085 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'88 2020-03-11T22:54:06.085 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'89 2020-03-11T22:54:06.085 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'90 2020-03-11T22:54:06.085 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'91 2020-03-11T22:54:06.085 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'92 2020-03-11T22:54:06.086 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'93 2020-03-11T22:54:06.086 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'94 2020-03-11T22:54:06.086 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'95 2020-03-11T22:54:06.086 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'96 2020-03-11T22:54:06.087 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'97 2020-03-11T22:54:06.087 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'98 2020-03-11T22:54:06.087 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'99 2020-03-11T22:54:06.087 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup version 34'100 2020-03-11T22:54:06.087 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'101 2020-03-11T22:54:06.087 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'102 2020-03-11T22:54:06.088 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'103 2020-03-11T22:54:06.088 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'104 2020-03-11T22:54:06.088 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'105 2020-03-11T22:54:06.088 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'106 2020-03-11T22:54:06.088 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'107 2020-03-11T22:54:06.088 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'108 2020-03-11T22:54:06.089 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'109 2020-03-11T22:54:06.089 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'110 2020-03-11T22:54:06.089 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'111 2020-03-11T22:54:06.089 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'112 2020-03-11T22:54:06.089 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'113 2020-03-11T22:54:06.089 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'114 2020-03-11T22:54:06.089 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'115 2020-03-11T22:54:06.090 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'116 2020-03-11T22:54:06.090 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'117 2020-03-11T22:54:06.090 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'118 2020-03-11T22:54:06.090 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'119 2020-03-11T22:54:06.090 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'120 2020-03-11T22:54:06.090 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'121 2020-03-11T22:54:06.090 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'122 2020-03-11T22:54:06.091 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'123 2020-03-11T22:54:06.091 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'124 2020-03-11T22:54:06.091 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'125 2020-03-11T22:54:06.091 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'126 2020-03-11T22:54:06.091 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'127 2020-03-11T22:54:06.091 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'128 2020-03-11T22:54:06.091 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'129 2020-03-11T22:54:06.092 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'130 2020-03-11T22:54:06.092 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'131 2020-03-11T22:54:06.092 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'132 2020-03-11T22:54:06.092 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'133 2020-03-11T22:54:06.092 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'134 2020-03-11T22:54:06.092 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'135 2020-03-11T22:54:06.093 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'136 2020-03-11T22:54:06.093 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'137 2020-03-11T22:54:06.093 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'138 2020-03-11T22:54:06.093 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'139 2020-03-11T22:54:06.093 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'140 2020-03-11T22:54:06.094 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'141 2020-03-11T22:54:06.094 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'142 2020-03-11T22:54:06.094 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'143 2020-03-11T22:54:06.094 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'144 2020-03-11T22:54:06.094 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'145 2020-03-11T22:54:06.094 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'146 2020-03-11T22:54:06.094 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'147 2020-03-11T22:54:06.095 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'148 2020-03-11T22:54:06.095 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'149 2020-03-11T22:54:06.095 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:01.829+0000 7f0f0d723700 20 copy_up_to/copy_after copy dup from log version 34'150 2020-03-11T22:54:06.095 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:93: _common_test: rm -f td/osd-backfill-recovery-log/result.log 2020-03-11T22:54:06.095 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:94: _common_test: '[' 0 '!=' 0 ']' 2020-03-11T22:54:06.095 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:34: run: teardown td/osd-backfill-recovery-log 2020-03-11T22:54:06.095 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-recovery-log 2020-03-11T22:54:06.096 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:54:06.096 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-recovery-log KILL 2020-03-11T22:54:06.096 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:54:06.096 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:54:06.096 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:54:06.096 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:54:06.097 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:54:06.097 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:54:06.097 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:54:06.097 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:54:06.097 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:54:06.097 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:54:06.098 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:54:06.098 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:54:06.098 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:54:06.098 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:54:06.098 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:54:06.098 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:54:06.098 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:54:06.099 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:54:06.099 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-recovery-log 2020-03-11T22:54:06.099 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:54:06.099 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:54:06.099 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76608 2020-03-11T22:54:06.099 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76608 2020-03-11T22:54:06.099 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:54:06.099 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:54:06.100 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:31: run: for func in $funcs 2020-03-11T22:54:06.100 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:32: run: setup td/osd-backfill-recovery-log 2020-03-11T22:54:06.100 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-recovery-log 2020-03-11T22:54:06.100 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-recovery-log 2020-03-11T22:54:06.100 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-recovery-log 2020-03-11T22:54:06.100 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:54:06.100 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-recovery-log KILL 2020-03-11T22:54:06.101 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:54:06.101 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:54:06.101 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:54:06.101 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:54:06.101 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:54:06.101 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:54:06.101 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:54:06.102 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:54:06.102 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:54:06.102 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:54:06.102 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:54:06.102 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:54:06.102 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:54:06.102 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:54:06.103 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:54:06.103 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:54:06.103 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:54:06.103 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:54:06.104 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-recovery-log 2020-03-11T22:54:06.104 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:54:06.104 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:54:06.104 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76608 2020-03-11T22:54:06.105 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76608 2020-03-11T22:54:06.105 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:54:06.105 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:54:06.105 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-recovery-log 2020-03-11T22:54:06.105 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-11T22:54:06.105 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:54:06.105 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76608 2020-03-11T22:54:06.106 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76608 2020-03-11T22:54:06.106 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-11T22:54:06.106 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-03-11T22:54:06.106 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-11T22:54:06.106 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-recovery-log 1' TERM HUP INT 2020-03-11T22:54:06.107 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:33: run: TEST_recovery_1 td/osd-backfill-recovery-log 2020-03-11T22:54:06.107 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:119: TEST_recovery_1: local dir=td/osd-backfill-recovery-log 2020-03-11T22:54:06.107 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:121: TEST_recovery_1: _common_test td/osd-backfill-recovery-log '--osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10' 40 0 40 2020-03-11T22:54:06.107 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:40: _common_test: local dir=td/osd-backfill-recovery-log 2020-03-11T22:54:06.107 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:41: _common_test: local 'extra_opts=--osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10' 2020-03-11T22:54:06.107 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:42: _common_test: local loglen=40 2020-03-11T22:54:06.107 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:43: _common_test: local dupslen=0 2020-03-11T22:54:06.108 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:44: _common_test: local objects=40 2020-03-11T22:54:06.108 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:45: _common_test: local moreobjects=0 2020-03-11T22:54:06.108 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:47: _common_test: local OSDS=6 2020-03-11T22:54:06.108 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:49: _common_test: run_mon td/osd-backfill-recovery-log a 2020-03-11T22:54:06.108 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-recovery-log 2020-03-11T22:54:06.108 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-11T22:54:06.108 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-11T22:54:06.109 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-11T22:54:06.109 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-recovery-log/a 2020-03-11T22:54:06.109 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-recovery-log/a --run-dir=td/osd-backfill-recovery-log 2020-03-11T22:54:06.316 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-11T22:54:06.316 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:54:06.316 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:54:06.316 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:54:06.316 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:54:06.316 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76608 2020-03-11T22:54:06.317 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76608/$cluster-$name.asok' 2020-03-11T22:54:06.317 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-recovery-log/a '--log-file=td/osd-backfill-recovery-log/$name.log' '--admin-socket=/tmp/ceph-asok.76608/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-recovery-log/log --run-dir=td/osd-backfill-recovery-log '--pid-file=td/osd-backfill-recovery-log/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-11T22:54:06.466 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-11T22:54:06.468 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-11T22:54:06.468 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:54:06.468 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:54:06.468 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-11T22:54:06.469 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-11T22:54:06.469 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:54:06.469 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:54:06.469 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:54:06.469 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:54:06.469 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:54:06.469 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76608 2020-03-11T22:54:06.470 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76608/ceph-mon.a.asok 2020-03-11T22:54:06.470 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:54:06.470 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.76608/ceph-mon.a.asok config get fsid 2020-03-11T22:54:06.564 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-11T22:54:06.565 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:54:06.565 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:54:06.565 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-11T22:54:06.565 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-11T22:54:06.565 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:54:06.566 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:54:06.566 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:54:06.566 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:54:06.566 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:54:06.566 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76608 2020-03-11T22:54:06.566 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76608/ceph-mon.a.asok 2020-03-11T22:54:06.567 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:54:06.567 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.76608/ceph-mon.a.asok config get mon_host 2020-03-11T22:54:06.656 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:50: _common_test: run_mgr td/osd-backfill-recovery-log x 2020-03-11T22:54:06.656 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/osd-backfill-recovery-log 2020-03-11T22:54:06.657 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-11T22:54:06.657 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-11T22:54:06.657 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-11T22:54:06.657 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/osd-backfill-recovery-log/x 2020-03-11T22:54:06.657 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-11T22:54:07.552 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-11T22:54:07.552 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:54:07.553 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:54:07.553 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:54:07.553 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:54:07.553 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76608 2020-03-11T22:54:07.554 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76608/$cluster-$name.asok' 2020-03-11T22:54:07.554 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:54:07.554 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-recovery-log/x '--log-file=td/osd-backfill-recovery-log/$name.log' '--admin-socket=/tmp/ceph-asok.76608/$cluster-$name.asok' --run-dir=td/osd-backfill-recovery-log '--pid-file=td/osd-backfill-recovery-log/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:54:07.577 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:51: _common_test: export CEPH_ARGS 2020-03-11T22:54:07.577 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:52: _common_test: export 'EXTRA_OPTS= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10' 2020-03-11T22:54:07.578 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:52: _common_test: EXTRA_OPTS=' --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10' 2020-03-11T22:54:07.578 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: expr 6 - 1 2020-03-11T22:54:07.579 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: seq 0 5 2020-03-11T22:54:07.582 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:54:07.583 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 0 2020-03-11T22:54:07.583 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:54:07.583 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:54:07.583 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-11T22:54:07.584 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:54:07.584 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/0 2020-03-11T22:54:07.584 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=7f0bc282-eebc-452f-ae5f-6537699866e7 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:54:07.584 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:54:07.584 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:54:07.585 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:54:07.585 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/0' 2020-03-11T22:54:07.585 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/0/journal' 2020-03-11T22:54:07.585 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:54:07.586 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10' 2020-03-11T22:54:07.586 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:54:07.586 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:54:07.586 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:54:07.586 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:54:07.586 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:54:07.586 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:54:07.587 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76608 2020-03-11T22:54:07.587 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76608/$cluster-$name.asok' 2020-03-11T22:54:07.587 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76608/$cluster-$name.asok' 2020-03-11T22:54:07.587 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:54:07.587 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:54:07.587 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:54:07.587 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:54:07.588 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:54:07.588 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:54:07.588 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:54:07.588 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:54:07.588 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:54:07.588 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:54:07.588 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/0 2020-03-11T22:54:07.589 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:54:07.589 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=068c054e-1071-4e65-9247-03d5658a75c6 2020-03-11T22:54:07.590 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 068c054e-1071-4e65-9247-03d5658a75c6' 2020-03-11T22:54:07.590 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:54:07.590 INFO:tasks.workunit.client.0.smithi059.stdout:add osd0 068c054e-1071-4e65-9247-03d5658a75c6 2020-03-11T22:54:07.597 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQAPbGleuHFxIxAA30ZKN0zK8rROx5k8nPM3zw== 2020-03-11T22:54:07.598 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQAPbGleuHFxIxAA30ZKN0zK8rROx5k8nPM3zw=="}' 2020-03-11T22:54:07.598 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 068c054e-1071-4e65-9247-03d5658a75c6 -i td/osd-backfill-recovery-log/0/new.json 2020-03-11T22:54:07.984 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:54:07.993 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/0/new.json 2020-03-11T22:54:07.994 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=7f0bc282-eebc-452f-ae5f-6537699866e7 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/0 --osd-journal=td/osd-backfill-recovery-log/0/journal --chdir= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76608/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAPbGleuHFxIxAA30ZKN0zK8rROx5k8nPM3zw== --osd-uuid 068c054e-1071-4e65-9247-03d5658a75c6 2020-03-11T22:54:08.020 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:54:08.017+0000 7fbe114ddec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:08.022 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:54:08.019+0000 7fbe114ddec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:08.023 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:54:08.020+0000 7fbe114ddec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:08.024 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:54:08.020+0000 7fbe114ddec0 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2020-03-11T22:54:08.024 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:54:08.020+0000 7fbe114ddec0 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2020-03-11T22:54:08.024 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:54:08.020+0000 7fbe114ddec0 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2020-03-11T22:54:08.074 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:54:08.071+0000 7fbe114ddec0 -1 bluestore(td/osd-backfill-recovery-log/0) _read_fsid unparsable uuid 2020-03-11T22:54:11.184 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/0/keyring 2020-03-11T22:54:11.185 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:54:11.185 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-11T22:54:11.186 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:54:11.186 INFO:tasks.workunit.client.0.smithi059.stdout:adding osd0 key to auth repository 2020-03-11T22:54:11.669 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.0 2020-03-11T22:54:11.670 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-11T22:54:11.670 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=7f0bc282-eebc-452f-ae5f-6537699866e7 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/0 --osd-journal=td/osd-backfill-recovery-log/0/journal --chdir= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76608/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:54:11.670 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:54:11.670 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:54:11.670 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:54:11.693 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:54:11.690+0000 7fad1f9d8ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:11.694 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:54:11.692+0000 7fad1f9d8ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:11.696 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:54:11.693+0000 7fad1f9d8ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:12.038 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-11T22:54:12.038 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:54:12.039 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:54:12.039 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:54:12.039 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:54:12.039 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:54:12.039 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:54:12.039 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:54:12.039 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:54:12.040 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:54:12.214 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:54:12.211+0000 7fad1f9d8ec0 -1 Falling back to public interface 2020-03-11T22:54:12.408 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:54:12.973 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:54:12.970+0000 7fad1f9d8ec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-11T22:54:13.409 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:54:13.410 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:54:13.410 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:54:13.410 INFO:tasks.workunit.client.0.smithi059.stdout:1 2020-03-11T22:54:13.410 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:54:13.411 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:54:13.787 INFO:tasks.workunit.client.0.smithi059.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/93123,v1:127.0.0.1:6803/93123] [v2:127.0.0.1:6804/93123,v1:127.0.0.1:6805/93123] exists,up 068c054e-1071-4e65-9247-03d5658a75c6 2020-03-11T22:54:13.787 INFO:tasks.workunit.client.0.smithi059.stdout:add osd1 ce7e18ed-4fe3-4925-bf87-f7768d9dbeee 2020-03-11T22:54:13.788 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:54:13.788 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:54:13.788 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:54:13.788 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:54:13.788 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 1 2020-03-11T22:54:13.788 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:54:13.788 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:54:13.789 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-11T22:54:13.789 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:54:13.789 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/1 2020-03-11T22:54:13.789 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=7f0bc282-eebc-452f-ae5f-6537699866e7 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:54:13.789 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:54:13.789 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:54:13.789 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:54:13.790 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/1' 2020-03-11T22:54:13.790 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/1/journal' 2020-03-11T22:54:13.790 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:54:13.790 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10' 2020-03-11T22:54:13.790 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:54:13.790 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:54:13.790 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:54:13.791 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:54:13.791 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:54:13.791 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:54:13.791 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76608 2020-03-11T22:54:13.791 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76608/$cluster-$name.asok' 2020-03-11T22:54:13.791 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76608/$cluster-$name.asok' 2020-03-11T22:54:13.792 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:54:13.792 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:54:13.792 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:54:13.792 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:54:13.792 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:54:13.792 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:54:13.792 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:54:13.793 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:54:13.793 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:54:13.793 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:54:13.793 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/1 2020-03-11T22:54:13.793 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:54:13.793 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=ce7e18ed-4fe3-4925-bf87-f7768d9dbeee 2020-03-11T22:54:13.793 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 ce7e18ed-4fe3-4925-bf87-f7768d9dbeee' 2020-03-11T22:54:13.794 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:54:13.802 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQAVbGleF4ijLxAA8Leb7sp7R83L4NmkFdUigA== 2020-03-11T22:54:13.802 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQAVbGleF4ijLxAA8Leb7sp7R83L4NmkFdUigA=="}' 2020-03-11T22:54:13.802 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new ce7e18ed-4fe3-4925-bf87-f7768d9dbeee -i td/osd-backfill-recovery-log/1/new.json 2020-03-11T22:54:14.373 INFO:tasks.workunit.client.0.smithi059.stdout:1 2020-03-11T22:54:14.383 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/1/new.json 2020-03-11T22:54:14.384 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=7f0bc282-eebc-452f-ae5f-6537699866e7 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/1 --osd-journal=td/osd-backfill-recovery-log/1/journal --chdir= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76608/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAVbGleF4ijLxAA8Leb7sp7R83L4NmkFdUigA== --osd-uuid ce7e18ed-4fe3-4925-bf87-f7768d9dbeee 2020-03-11T22:54:14.410 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:54:14.407+0000 7f1466c1cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:14.412 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:54:14.409+0000 7f1466c1cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:14.413 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:54:14.411+0000 7f1466c1cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:14.414 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:54:14.411+0000 7f1466c1cec0 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2020-03-11T22:54:14.414 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:54:14.411+0000 7f1466c1cec0 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2020-03-11T22:54:14.414 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:54:14.411+0000 7f1466c1cec0 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2020-03-11T22:54:14.546 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:54:14.543+0000 7f1466c1cec0 -1 bluestore(td/osd-backfill-recovery-log/1) _read_fsid unparsable uuid 2020-03-11T22:54:17.732 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/1/keyring 2020-03-11T22:54:17.733 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:54:17.734 INFO:tasks.workunit.client.0.smithi059.stdout:adding osd1 key to auth repository 2020-03-11T22:54:17.734 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-03-11T22:54:17.734 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:54:18.217 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.1 2020-03-11T22:54:18.217 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-03-11T22:54:18.217 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:54:18.217 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:54:18.217 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:54:18.218 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=7f0bc282-eebc-452f-ae5f-6537699866e7 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/1 --osd-journal=td/osd-backfill-recovery-log/1/journal --chdir= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76608/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:54:18.240 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:54:18.237+0000 7fd03d2acec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:18.242 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:54:18.239+0000 7fd03d2acec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:18.244 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:54:18.241+0000 7fd03d2acec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:18.578 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-03-11T22:54:18.579 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:54:18.579 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:54:18.579 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:54:18.579 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:54:18.579 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:54:18.579 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:54:18.580 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:54:18.580 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:54:18.580 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:54:18.761 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:54:18.758+0000 7fd03d2acec0 -1 Falling back to public interface 2020-03-11T22:54:18.946 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:54:19.530 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:54:19.527+0000 7fd03d2acec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-11T22:54:19.948 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:54:19.948 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:54:19.948 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:54:19.948 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:54:19.949 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:54:19.949 INFO:tasks.workunit.client.0.smithi059.stdout:1 2020-03-11T22:54:20.394 INFO:tasks.workunit.client.0.smithi059.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/93478,v1:127.0.0.1:6811/93478] [v2:127.0.0.1:6812/93478,v1:127.0.0.1:6813/93478] exists,up ce7e18ed-4fe3-4925-bf87-f7768d9dbeee 2020-03-11T22:54:20.394 INFO:tasks.workunit.client.0.smithi059.stdout:add osd2 9687c573-adfb-463b-b053-011e6be5bab6 2020-03-11T22:54:20.394 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:54:20.395 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:54:20.395 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:54:20.395 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:54:20.395 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 2 2020-03-11T22:54:20.395 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:54:20.395 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:54:20.395 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-03-11T22:54:20.396 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:54:20.396 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/2 2020-03-11T22:54:20.396 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=7f0bc282-eebc-452f-ae5f-6537699866e7 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:54:20.396 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:54:20.396 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:54:20.396 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:54:20.396 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/2' 2020-03-11T22:54:20.397 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/2/journal' 2020-03-11T22:54:20.397 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:54:20.397 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10' 2020-03-11T22:54:20.397 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:54:20.397 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:54:20.397 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:54:20.397 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:54:20.398 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:54:20.398 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:54:20.398 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76608 2020-03-11T22:54:20.398 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76608/$cluster-$name.asok' 2020-03-11T22:54:20.398 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76608/$cluster-$name.asok' 2020-03-11T22:54:20.398 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:54:20.398 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:54:20.399 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:54:20.399 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:54:20.399 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:54:20.399 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:54:20.399 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:54:20.399 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:54:20.399 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:54:20.400 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:54:20.400 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/2 2020-03-11T22:54:20.400 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:54:20.400 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=9687c573-adfb-463b-b053-011e6be5bab6 2020-03-11T22:54:20.400 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 9687c573-adfb-463b-b053-011e6be5bab6' 2020-03-11T22:54:20.400 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:54:20.408 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQAcbGleno8mGBAAphgT+ujNYGFouobwpUTnyg== 2020-03-11T22:54:20.408 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQAcbGleno8mGBAAphgT+ujNYGFouobwpUTnyg=="}' 2020-03-11T22:54:20.408 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 9687c573-adfb-463b-b053-011e6be5bab6 -i td/osd-backfill-recovery-log/2/new.json 2020-03-11T22:54:20.965 INFO:tasks.workunit.client.0.smithi059.stdout:2 2020-03-11T22:54:20.975 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/2/new.json 2020-03-11T22:54:20.976 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=7f0bc282-eebc-452f-ae5f-6537699866e7 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/2 --osd-journal=td/osd-backfill-recovery-log/2/journal --chdir= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76608/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAcbGleno8mGBAAphgT+ujNYGFouobwpUTnyg== --osd-uuid 9687c573-adfb-463b-b053-011e6be5bab6 2020-03-11T22:54:21.001 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:54:20.998+0000 7f9641fa2ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:21.036 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:54:21.033+0000 7f9641fa2ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:21.037 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:54:21.034+0000 7f9641fa2ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:21.037 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:54:21.034+0000 7f9641fa2ec0 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2020-03-11T22:54:21.038 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:54:21.034+0000 7f9641fa2ec0 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2020-03-11T22:54:21.038 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:54:21.034+0000 7f9641fa2ec0 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2020-03-11T22:54:21.161 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:54:21.158+0000 7f9641fa2ec0 -1 bluestore(td/osd-backfill-recovery-log/2) _read_fsid unparsable uuid 2020-03-11T22:54:24.570 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/2/keyring 2020-03-11T22:54:24.570 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:54:24.571 INFO:tasks.workunit.client.0.smithi059.stdout:adding osd2 key to auth repository 2020-03-11T22:54:24.572 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-03-11T22:54:24.572 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:54:25.056 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.2 2020-03-11T22:54:25.056 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-03-11T22:54:25.057 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=7f0bc282-eebc-452f-ae5f-6537699866e7 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/2 --osd-journal=td/osd-backfill-recovery-log/2/journal --chdir= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76608/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:54:25.057 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:54:25.057 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:54:25.057 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:54:25.079 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:54:25.076+0000 7f645b5c4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:25.081 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:54:25.078+0000 7f645b5c4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:25.083 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:54:25.080+0000 7f645b5c4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:25.447 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-03-11T22:54:25.448 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:54:25.448 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:54:25.448 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:54:25.448 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:54:25.449 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:54:25.449 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:54:25.449 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:54:25.449 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:54:25.450 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:54:25.613 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:54:25.610+0000 7f645b5c4ec0 -1 Falling back to public interface 2020-03-11T22:54:25.829 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:54:26.397 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:54:26.394+0000 7f645b5c4ec0 -1 osd.2 0 log_to_monitors {default=true} 2020-03-11T22:54:26.830 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:54:26.831 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:54:26.831 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:54:26.831 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:54:26.831 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:54:26.831 INFO:tasks.workunit.client.0.smithi059.stdout:1 2020-03-11T22:54:27.224 INFO:tasks.workunit.client.0.smithi059.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/93833,v1:127.0.0.1:6819/93833] [v2:127.0.0.1:6820/93833,v1:127.0.0.1:6821/93833] exists,up 9687c573-adfb-463b-b053-011e6be5bab6 2020-03-11T22:54:27.224 INFO:tasks.workunit.client.0.smithi059.stdout:add osd3 bb5c53b5-4c86-41d4-bc26-7c175948f542 2020-03-11T22:54:27.225 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:54:27.225 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:54:27.225 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:54:27.225 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:54:27.225 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 3 2020-03-11T22:54:27.226 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:54:27.226 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:54:27.226 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=3 2020-03-11T22:54:27.226 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:54:27.226 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/3 2020-03-11T22:54:27.226 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=7f0bc282-eebc-452f-ae5f-6537699866e7 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:54:27.226 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:54:27.227 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:54:27.227 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:54:27.227 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/3' 2020-03-11T22:54:27.227 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/3/journal' 2020-03-11T22:54:27.227 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:54:27.227 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10' 2020-03-11T22:54:27.227 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:54:27.228 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:54:27.228 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:54:27.228 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:54:27.228 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:54:27.228 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:54:27.228 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76608 2020-03-11T22:54:27.228 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76608/$cluster-$name.asok' 2020-03-11T22:54:27.229 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76608/$cluster-$name.asok' 2020-03-11T22:54:27.229 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:54:27.229 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:54:27.229 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:54:27.229 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:54:27.229 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:54:27.229 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:54:27.230 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:54:27.230 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:54:27.230 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:54:27.230 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:54:27.230 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/3 2020-03-11T22:54:27.230 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:54:27.230 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=bb5c53b5-4c86-41d4-bc26-7c175948f542 2020-03-11T22:54:27.231 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd3 bb5c53b5-4c86-41d4-bc26-7c175948f542' 2020-03-11T22:54:27.231 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:54:27.238 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQAjbGleD9oFDhAAIVpfdFUE1lfieS3U/JNfYg== 2020-03-11T22:54:27.238 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQAjbGleD9oFDhAAIVpfdFUE1lfieS3U/JNfYg=="}' 2020-03-11T22:54:27.238 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new bb5c53b5-4c86-41d4-bc26-7c175948f542 -i td/osd-backfill-recovery-log/3/new.json 2020-03-11T22:54:27.816 INFO:tasks.workunit.client.0.smithi059.stdout:3 2020-03-11T22:54:27.826 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/3/new.json 2020-03-11T22:54:27.827 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 3 --fsid=7f0bc282-eebc-452f-ae5f-6537699866e7 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/3 --osd-journal=td/osd-backfill-recovery-log/3/journal --chdir= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76608/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAjbGleD9oFDhAAIVpfdFUE1lfieS3U/JNfYg== --osd-uuid bb5c53b5-4c86-41d4-bc26-7c175948f542 2020-03-11T22:54:27.852 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:54:27.850+0000 7fed09f65ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:27.918 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:54:27.916+0000 7fed09f65ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:27.920 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:54:27.917+0000 7fed09f65ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:27.920 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:54:27.917+0000 7fed09f65ec0 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2020-03-11T22:54:27.920 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:54:27.917+0000 7fed09f65ec0 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2020-03-11T22:54:27.920 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:54:27.917+0000 7fed09f65ec0 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2020-03-11T22:54:27.987 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:54:27.984+0000 7fed09f65ec0 -1 bluestore(td/osd-backfill-recovery-log/3) _read_fsid unparsable uuid 2020-03-11T22:54:31.098 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/3/keyring 2020-03-11T22:54:31.098 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:54:31.099 INFO:tasks.workunit.client.0.smithi059.stdout:adding osd3 key to auth repository 2020-03-11T22:54:31.100 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd3 key to auth repository 2020-03-11T22:54:31.100 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:54:31.585 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.3 2020-03-11T22:54:31.585 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.3 2020-03-11T22:54:31.585 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 3 --fsid=7f0bc282-eebc-452f-ae5f-6537699866e7 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/3 --osd-journal=td/osd-backfill-recovery-log/3/journal --chdir= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76608/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:54:31.585 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:54:31.586 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:54:31.586 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:54:31.608 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:54:31.605+0000 7f15371beec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:31.610 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:54:31.607+0000 7f15371beec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:31.611 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:54:31.608+0000 7f15371beec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:31.951 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 3 2020-03-11T22:54:31.951 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:54:31.951 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=3 2020-03-11T22:54:31.952 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:54:31.952 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:54:31.952 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:54:31.952 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:54:31.952 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:54:31.952 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:54:31.952 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:54:32.128 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:54:32.125+0000 7f15371beec0 -1 Falling back to public interface 2020-03-11T22:54:32.342 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:54:32.888 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:54:32.885+0000 7f15371beec0 -1 osd.3 0 log_to_monitors {default=true} 2020-03-11T22:54:33.343 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:54:33.344 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:54:33.344 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:54:33.344 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:54:33.344 INFO:tasks.workunit.client.0.smithi059.stdout:1 2020-03-11T22:54:33.345 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:54:33.736 INFO:tasks.workunit.client.0.smithi059.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/94188,v1:127.0.0.1:6827/94188] [v2:127.0.0.1:6828/94188,v1:127.0.0.1:6829/94188] exists,up bb5c53b5-4c86-41d4-bc26-7c175948f542 2020-03-11T22:54:33.736 INFO:tasks.workunit.client.0.smithi059.stdout:add osd4 fb67b3fd-f346-4105-9080-acbab0946f9b 2020-03-11T22:54:33.736 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:54:33.736 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:54:33.737 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:54:33.737 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:54:33.737 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 4 2020-03-11T22:54:33.737 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:54:33.737 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:54:33.737 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=4 2020-03-11T22:54:33.738 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:54:33.738 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/4 2020-03-11T22:54:33.738 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=7f0bc282-eebc-452f-ae5f-6537699866e7 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:54:33.738 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:54:33.738 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:54:33.738 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:54:33.738 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/4' 2020-03-11T22:54:33.739 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/4/journal' 2020-03-11T22:54:33.739 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:54:33.739 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10' 2020-03-11T22:54:33.739 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:54:33.739 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:54:33.739 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:54:33.739 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:54:33.739 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:54:33.740 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:54:33.740 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76608 2020-03-11T22:54:33.740 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76608/$cluster-$name.asok' 2020-03-11T22:54:33.740 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76608/$cluster-$name.asok' 2020-03-11T22:54:33.740 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:54:33.740 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:54:33.740 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:54:33.741 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:54:33.741 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:54:33.741 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:54:33.741 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:54:33.741 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:54:33.741 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:54:33.741 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:54:33.742 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/4 2020-03-11T22:54:33.742 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:54:33.742 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=fb67b3fd-f346-4105-9080-acbab0946f9b 2020-03-11T22:54:33.742 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd4 fb67b3fd-f346-4105-9080-acbab0946f9b' 2020-03-11T22:54:33.742 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:54:33.749 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQApbGleGgGELBAAU6XXt+CWUjasUUdTEwoECg== 2020-03-11T22:54:33.750 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQApbGleGgGELBAAU6XXt+CWUjasUUdTEwoECg=="}' 2020-03-11T22:54:33.750 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new fb67b3fd-f346-4105-9080-acbab0946f9b -i td/osd-backfill-recovery-log/4/new.json 2020-03-11T22:54:34.341 INFO:tasks.workunit.client.0.smithi059.stdout:4 2020-03-11T22:54:34.351 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/4/new.json 2020-03-11T22:54:34.352 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 4 --fsid=7f0bc282-eebc-452f-ae5f-6537699866e7 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/4 --osd-journal=td/osd-backfill-recovery-log/4/journal --chdir= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76608/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQApbGleGgGELBAAU6XXt+CWUjasUUdTEwoECg== --osd-uuid fb67b3fd-f346-4105-9080-acbab0946f9b 2020-03-11T22:54:34.377 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:54:34.374+0000 7fd5d86a7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:34.461 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:54:34.439+0000 7fd5d86a7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:34.464 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:54:34.461+0000 7fd5d86a7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:34.464 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:54:34.461+0000 7fd5d86a7ec0 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2020-03-11T22:54:34.464 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:54:34.461+0000 7fd5d86a7ec0 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2020-03-11T22:54:34.465 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:54:34.461+0000 7fd5d86a7ec0 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2020-03-11T22:54:34.536 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:54:34.534+0000 7fd5d86a7ec0 -1 bluestore(td/osd-backfill-recovery-log/4) _read_fsid unparsable uuid 2020-03-11T22:54:37.884 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/4/keyring 2020-03-11T22:54:37.884 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:54:37.885 INFO:tasks.workunit.client.0.smithi059.stdout:adding osd4 key to auth repository 2020-03-11T22:54:37.885 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd4 key to auth repository 2020-03-11T22:54:37.885 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:54:38.370 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.4 2020-03-11T22:54:38.370 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.4 2020-03-11T22:54:38.370 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 4 --fsid=7f0bc282-eebc-452f-ae5f-6537699866e7 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/4 --osd-journal=td/osd-backfill-recovery-log/4/journal --chdir= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76608/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:54:38.370 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:54:38.370 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:54:38.371 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:54:38.393 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:54:38.390+0000 7fd676a0aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:38.395 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:54:38.392+0000 7fd676a0aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:38.395 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:54:38.393+0000 7fd676a0aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:38.732 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 4 2020-03-11T22:54:38.732 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:54:38.732 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=4 2020-03-11T22:54:38.732 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:54:38.732 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:54:38.732 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:54:38.732 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:54:38.733 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:54:38.733 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.4 up' 2020-03-11T22:54:38.733 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:54:38.912 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:54:38.909+0000 7fd676a0aec0 -1 Falling back to public interface 2020-03-11T22:54:39.124 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:54:39.697 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:54:39.694+0000 7fd676a0aec0 -1 osd.4 0 log_to_monitors {default=true} 2020-03-11T22:54:40.126 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:54:40.126 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:54:40.126 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:54:40.126 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:54:40.126 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.4 up' 2020-03-11T22:54:40.127 INFO:tasks.workunit.client.0.smithi059.stdout:1 2020-03-11T22:54:40.530 INFO:tasks.workunit.client.0.smithi059.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/94551,v1:127.0.0.1:6835/94551] [v2:127.0.0.1:6836/94551,v1:127.0.0.1:6837/94551] exists,up fb67b3fd-f346-4105-9080-acbab0946f9b 2020-03-11T22:54:40.530 INFO:tasks.workunit.client.0.smithi059.stdout:add osd5 4ff0d6c9-c859-4fb6-9fcc-53cc6756d768 2020-03-11T22:54:40.530 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:54:40.531 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:54:40.531 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:54:40.531 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:54:40.531 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 5 2020-03-11T22:54:40.531 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:54:40.531 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:54:40.531 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=5 2020-03-11T22:54:40.532 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:54:40.532 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/5 2020-03-11T22:54:40.532 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=7f0bc282-eebc-452f-ae5f-6537699866e7 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:54:40.532 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:54:40.532 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:54:40.532 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:54:40.532 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/5' 2020-03-11T22:54:40.533 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/5/journal' 2020-03-11T22:54:40.533 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:54:40.533 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10' 2020-03-11T22:54:40.533 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:54:40.533 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:54:40.533 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:54:40.533 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:54:40.534 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:54:40.534 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:54:40.534 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76608 2020-03-11T22:54:40.534 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76608/$cluster-$name.asok' 2020-03-11T22:54:40.534 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76608/$cluster-$name.asok' 2020-03-11T22:54:40.534 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:54:40.534 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:54:40.535 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:54:40.535 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:54:40.535 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:54:40.535 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:54:40.535 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:54:40.535 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:54:40.535 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:54:40.536 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:54:40.536 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/5 2020-03-11T22:54:40.536 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:54:40.536 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=4ff0d6c9-c859-4fb6-9fcc-53cc6756d768 2020-03-11T22:54:40.536 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd5 4ff0d6c9-c859-4fb6-9fcc-53cc6756d768' 2020-03-11T22:54:40.536 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:54:40.544 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQAwbGleIkZDIBAAdxXBDeB0cqSnwvmm6ySjKA== 2020-03-11T22:54:40.544 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQAwbGleIkZDIBAAdxXBDeB0cqSnwvmm6ySjKA=="}' 2020-03-11T22:54:40.544 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 4ff0d6c9-c859-4fb6-9fcc-53cc6756d768 -i td/osd-backfill-recovery-log/5/new.json 2020-03-11T22:54:41.100 INFO:tasks.workunit.client.0.smithi059.stdout:5 2020-03-11T22:54:41.110 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/5/new.json 2020-03-11T22:54:41.111 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 5 --fsid=7f0bc282-eebc-452f-ae5f-6537699866e7 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/5 --osd-journal=td/osd-backfill-recovery-log/5/journal --chdir= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76608/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAwbGleIkZDIBAAdxXBDeB0cqSnwvmm6ySjKA== --osd-uuid 4ff0d6c9-c859-4fb6-9fcc-53cc6756d768 2020-03-11T22:54:41.137 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:54:41.134+0000 7f316acefec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:41.162 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:54:41.159+0000 7f316acefec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:41.164 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:54:41.161+0000 7f316acefec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:41.165 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:54:41.162+0000 7f316acefec0 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2020-03-11T22:54:41.165 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:54:41.162+0000 7f316acefec0 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2020-03-11T22:54:41.165 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:54:41.162+0000 7f316acefec0 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2020-03-11T22:54:41.270 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:54:41.268+0000 7f316acefec0 -1 bluestore(td/osd-backfill-recovery-log/5) _read_fsid unparsable uuid 2020-03-11T22:54:44.381 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/5/keyring 2020-03-11T22:54:44.381 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:54:44.382 INFO:tasks.workunit.client.0.smithi059.stdout:adding osd5 key to auth repository 2020-03-11T22:54:44.383 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd5 key to auth repository 2020-03-11T22:54:44.383 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:54:44.868 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.5 2020-03-11T22:54:44.868 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.5 2020-03-11T22:54:44.868 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 5 --fsid=7f0bc282-eebc-452f-ae5f-6537699866e7 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/5 --osd-journal=td/osd-backfill-recovery-log/5/journal --chdir= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76608/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:54:44.868 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:54:44.868 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:54:44.869 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:54:44.891 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:54:44.887+0000 7fb28b7b3ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:44.892 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:54:44.889+0000 7fb28b7b3ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:44.894 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:54:44.890+0000 7fb28b7b3ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:45.231 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 5 2020-03-11T22:54:45.232 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:54:45.232 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=5 2020-03-11T22:54:45.232 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:54:45.232 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:54:45.232 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:54:45.232 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:54:45.233 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:54:45.233 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.5 up' 2020-03-11T22:54:45.233 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:54:45.409 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:54:45.405+0000 7fb28b7b3ec0 -1 Falling back to public interface 2020-03-11T22:54:45.603 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:54:46.190 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:54:46.185+0000 7fb28b7b3ec0 -1 osd.5 0 log_to_monitors {default=true} 2020-03-11T22:54:46.605 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:54:46.605 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:54:46.605 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:54:46.605 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:54:46.605 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.5 up' 2020-03-11T22:54:46.606 INFO:tasks.workunit.client.0.smithi059.stdout:1 2020-03-11T22:54:46.983 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:54:47.985 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:54:47.985 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:54:47.985 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 2 2020-03-11T22:54:47.985 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:54:47.986 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.5 up' 2020-03-11T22:54:47.986 INFO:tasks.workunit.client.0.smithi059.stdout:2 2020-03-11T22:54:48.356 INFO:tasks.workunit.client.0.smithi059.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/94906,v1:127.0.0.1:6843/94906] [v2:127.0.0.1:6844/94906,v1:127.0.0.1:6845/94906] exists,up 4ff0d6c9-c859-4fb6-9fcc-53cc6756d768 2020-03-11T22:54:48.357 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:54:48.357 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:54:48.357 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:54:48.357 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:59: _common_test: create_pool test 1 1 2020-03-11T22:54:48.357 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test 1 1 2020-03-11T22:54:48.977 INFO:tasks.workunit.client.0.smithi059.stderr:pool 'test' already exists 2020-03-11T22:54:48.985 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:54:49.987 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: seq 1 40 2020-03-11T22:54:49.988 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:49.988 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-1 /etc/passwd 2020-03-11T22:54:50.038 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:50.039 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-2 /etc/passwd 2020-03-11T22:54:50.105 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:50.105 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-3 /etc/passwd 2020-03-11T22:54:50.186 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:50.187 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-4 /etc/passwd 2020-03-11T22:54:50.234 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:50.234 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-5 /etc/passwd 2020-03-11T22:54:50.294 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:50.294 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-6 /etc/passwd 2020-03-11T22:54:50.330 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:50.330 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-7 /etc/passwd 2020-03-11T22:54:50.403 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:50.404 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-8 /etc/passwd 2020-03-11T22:54:50.504 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:50.504 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-9 /etc/passwd 2020-03-11T22:54:50.574 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:50.574 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-10 /etc/passwd 2020-03-11T22:54:50.654 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:50.654 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-11 /etc/passwd 2020-03-11T22:54:50.709 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:50.709 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-12 /etc/passwd 2020-03-11T22:54:50.766 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:50.766 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-13 /etc/passwd 2020-03-11T22:54:50.801 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:50.801 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-14 /etc/passwd 2020-03-11T22:54:50.905 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:50.905 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-15 /etc/passwd 2020-03-11T22:54:51.005 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:51.006 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-16 /etc/passwd 2020-03-11T22:54:51.043 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:51.043 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-17 /etc/passwd 2020-03-11T22:54:51.076 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:51.077 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-18 /etc/passwd 2020-03-11T22:54:51.240 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:51.240 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-19 /etc/passwd 2020-03-11T22:54:51.285 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:51.285 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-20 /etc/passwd 2020-03-11T22:54:51.318 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:51.318 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-21 /etc/passwd 2020-03-11T22:54:51.457 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:51.457 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-22 /etc/passwd 2020-03-11T22:54:51.532 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:51.532 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-23 /etc/passwd 2020-03-11T22:54:51.568 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:51.569 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-24 /etc/passwd 2020-03-11T22:54:51.601 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:51.602 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-25 /etc/passwd 2020-03-11T22:54:51.745 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:51.745 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-26 /etc/passwd 2020-03-11T22:54:51.781 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:51.781 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-27 /etc/passwd 2020-03-11T22:54:51.816 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:51.816 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-28 /etc/passwd 2020-03-11T22:54:51.916 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:51.917 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-29 /etc/passwd 2020-03-11T22:54:52.017 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:52.017 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-30 /etc/passwd 2020-03-11T22:54:52.056 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:52.056 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-31 /etc/passwd 2020-03-11T22:54:52.117 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:52.117 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-32 /etc/passwd 2020-03-11T22:54:52.202 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:52.202 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-33 /etc/passwd 2020-03-11T22:54:52.268 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:52.269 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-34 /etc/passwd 2020-03-11T22:54:52.302 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:52.302 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-35 /etc/passwd 2020-03-11T22:54:52.376 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:52.376 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-36 /etc/passwd 2020-03-11T22:54:52.460 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:52.460 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-37 /etc/passwd 2020-03-11T22:54:52.497 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:52.498 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-38 /etc/passwd 2020-03-11T22:54:52.559 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:52.559 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-39 /etc/passwd 2020-03-11T22:54:52.652 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:52.652 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-40 /etc/passwd 2020-03-11T22:54:52.692 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:67: _common_test: ceph pg dump pgs --format=json 2020-03-11T22:54:52.693 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:67: _common_test: jq '.pg_stats[0].up[]' 2020-03-11T22:54:53.035 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2020-03-11T22:54:53.043 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:67: _common_test: ceph osd out 1 0 2 2020-03-11T22:54:53.616 INFO:tasks.workunit.client.0.smithi059.stderr:osd.1 is already out. osd.0 is already out. osd.2 is already out. 2020-03-11T22:54:53.624 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:68: _common_test: '[' 0 '!=' 0 ']' 2020-03-11T22:54:53.624 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:74: _common_test: sleep 1 2020-03-11T22:54:54.625 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:75: _common_test: wait_for_clean 2020-03-11T22:54:54.626 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:54:54.626 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:54:54.626 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:54:54.626 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:54:54.626 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:54:54.627 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:54:54.627 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:54:54.627 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:54:54.628 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:54:54.628 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:54:54.734 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:54:54.734 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:54:54.734 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:54:54.734 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:54:54.734 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:54:55.106 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:54:55.107 INFO:tasks.workunit.client.0.smithi059.stderr:1 2020-03-11T22:54:55.107 INFO:tasks.workunit.client.0.smithi059.stderr:2 2020-03-11T22:54:55.107 INFO:tasks.workunit.client.0.smithi059.stderr:3 2020-03-11T22:54:55.107 INFO:tasks.workunit.client.0.smithi059.stderr:4 2020-03-11T22:54:55.107 INFO:tasks.workunit.client.0.smithi059.stderr:5' 2020-03-11T22:54:55.107 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:54:55.107 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:55.108 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:54:55.236 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836491 2020-03-11T22:54:55.236 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836491' 2020-03-11T22:54:55.236 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:55.236 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:54:55.365 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672969 2020-03-11T22:54:55.365 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836491 1-42949672969' 2020-03-11T22:54:55.365 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:55.365 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:54:55.494 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509448 2020-03-11T22:54:55.494 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836491 1-42949672969 2-64424509448' 2020-03-11T22:54:55.494 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:55.494 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:54:55.623 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=85899345927 2020-03-11T22:54:55.623 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836491 1-42949672969 2-64424509448 3-85899345927' 2020-03-11T22:54:55.623 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:55.624 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:54:55.753 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=107374182405 2020-03-11T22:54:55.753 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836491 1-42949672969 2-64424509448 3-85899345927 4-107374182405' 2020-03-11T22:54:55.753 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:55.754 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2020-03-11T22:54:55.883 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=128849018884 2020-03-11T22:54:55.883 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836491 1-42949672969 2-64424509448 3-85899345927 4-107374182405 5-128849018884' 2020-03-11T22:54:55.884 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:54:55.884 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836491 2020-03-11T22:54:55.884 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:54:55.886 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:54:55.886 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836491 2020-03-11T22:54:55.886 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:54:55.887 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836491 2020-03-11T22:54:55.887 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2020-03-11T22:54:55.887 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:54:55.887 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.0 seq 21474836491 2020-03-11T22:54:56.250 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836489 -lt 21474836491 2020-03-11T22:54:56.250 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:54:57.252 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:54:57.252 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:54:57.633 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836491 -lt 21474836491 2020-03-11T22:54:57.633 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:54:57.634 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672969 2020-03-11T22:54:57.634 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:54:57.634 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:54:57.634 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672969 2020-03-11T22:54:57.634 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:54:57.634 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672969 2020-03-11T22:54:57.634 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2020-03-11T22:54:57.635 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:54:57.635 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.1 seq 42949672969 2020-03-11T22:54:58.000 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672970 -lt 42949672969 2020-03-11T22:54:58.000 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:54:58.000 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509448 2020-03-11T22:54:58.001 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:54:58.001 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:54:58.001 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509448 2020-03-11T22:54:58.002 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:54:58.002 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509448 2020-03-11T22:54:58.003 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2020-03-11T22:54:58.003 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:54:58.003 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.2 seq 64424509448 2020-03-11T22:54:58.375 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509448 -lt 64424509448 2020-03-11T22:54:58.375 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:54:58.376 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-85899345927 2020-03-11T22:54:58.376 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:54:58.376 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:54:58.377 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-85899345927 2020-03-11T22:54:58.377 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:54:58.378 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=85899345927 2020-03-11T22:54:58.378 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 85899345927' 2020-03-11T22:54:58.378 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:54:58.379 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.3 seq 85899345927 2020-03-11T22:54:58.842 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 85899345927 -lt 85899345927 2020-03-11T22:54:58.842 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:54:58.843 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-107374182405 2020-03-11T22:54:58.843 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:54:58.844 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:54:58.844 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-107374182405 2020-03-11T22:54:58.844 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:54:58.845 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=107374182405 2020-03-11T22:54:58.845 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 107374182405' 2020-03-11T22:54:58.845 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:54:58.846 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.4 seq 107374182405 2020-03-11T22:54:59.212 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 107374182406 -lt 107374182405 2020-03-11T22:54:59.212 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:54:59.212 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 5-128849018884 2020-03-11T22:54:59.212 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:54:59.213 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=5 2020-03-11T22:54:59.213 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 5-128849018884 2020-03-11T22:54:59.214 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:54:59.215 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=128849018884 2020-03-11T22:54:59.215 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.5 seq 128849018884' 2020-03-11T22:54:59.215 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 5 2020-03-11T22:54:59.215 INFO:tasks.workunit.client.0.smithi059.stdout:waiting osd.5 seq 128849018884 2020-03-11T22:54:59.581 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 128849018885 -lt 128849018884 2020-03-11T22:54:59.581 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:54:59.582 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:59.582 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:55:00.053 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:55:00.053 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:55:00.053 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:55:00.053 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:55:00.054 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:55:00.054 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:55:00.054 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:55:00.054 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:55:00.409 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:55:00.410 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:55:00.410 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:55:00.410 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:55:00.884 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:55:00.884 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:55:00.884 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:55:00.884 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:55:00.884 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:55:00.885 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:55:00.985 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:55:00.986 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:55:00.986 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:55:00.986 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:55:00.986 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:55:00.986 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:55:00.986 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:55:00.987 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:55:01.337 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:55:01.337 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:55:01.337 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:55:01.338 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:55:01.815 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:55:01.815 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:55:01.815 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:55:01.816 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:55:01.816 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:55:01.816 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:55:01.816 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:55:01.816 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:55:01.816 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:55:02.290 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=6898 2020-03-11T22:55:02.290 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 6898 '!=' null 2020-03-11T22:55:02.290 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:55:02.290 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:55:02.291 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:55:02.391 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:55:02.391 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:55:02.391 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:55:02.391 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:55:02.391 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:55:02.392 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:55:02.392 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:55:02.392 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:55:02.742 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:55:02.743 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:55:02.743 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:55:02.743 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:55:03.210 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:55:03.211 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:55:03.211 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:55:03.211 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:55:03.211 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:55:03.211 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:55:03.211 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:55:03.212 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:55:03.212 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:55:03.706 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=26038 2020-03-11T22:55:03.706 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 26038 '!=' null 2020-03-11T22:55:03.706 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:55:03.706 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:55:03.707 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:55:03.806 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:55:03.807 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:55:03.807 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:55:03.808 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:55:03.808 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:55:03.808 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:55:03.808 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:55:03.808 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:55:04.156 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:55:04.157 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:55:04.157 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:55:04.157 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:55:04.640 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:55:04.640 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:55:04.641 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:55:04.641 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:55:04.641 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:55:04.641 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:55:04.641 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:55:04.641 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:55:04.642 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:55:05.115 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=26038 2020-03-11T22:55:05.115 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 26038 '!=' null 2020-03-11T22:55:05.115 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:55:05.116 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:55:05.116 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:55:05.216 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:55:05.217 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:55:05.217 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:55:05.217 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:55:05.217 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:55:05.217 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:55:05.217 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:55:05.217 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:55:05.568 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:55:05.568 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:55:05.569 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:55:05.569 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:55:06.040 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:55:06.040 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:55:06.040 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:55:06.040 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:55:06.041 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:55:06.041 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:55:06.041 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:55:06.041 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:55:06.041 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:55:06.530 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=22677 2020-03-11T22:55:06.531 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 22677 '!=' null 2020-03-11T22:55:06.531 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:55:06.531 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:55:06.531 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:55:06.632 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:55:06.632 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:55:06.632 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:55:06.632 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:55:06.632 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:55:06.632 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:55:06.632 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:55:06.633 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:55:06.990 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:55:06.990 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:55:06.991 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:55:06.991 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:55:07.468 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:55:07.468 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:55:07.468 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:55:07.468 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:55:07.468 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:55:07.469 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:55:07.469 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:55:07.469 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:55:07.469 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:55:07.945 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=22998 2020-03-11T22:55:07.946 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 22998 '!=' null 2020-03-11T22:55:07.946 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:55:07.946 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:55:07.946 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:55:08.046 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:55:08.046 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:55:08.047 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:55:08.047 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:55:08.047 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:55:08.048 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:55:08.048 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:55:08.048 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:55:08.400 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:55:08.400 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:55:08.400 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:55:08.400 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:55:08.875 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:55:08.875 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:55:08.875 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:55:08.875 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:77: _common_test: ceph pg dump pgs --format=json 2020-03-11T22:55:08.875 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:77: _common_test: jq '.pg_stats[0].up_primary' 2020-03-11T22:55:09.219 INFO:tasks.workunit.client.0.smithi059.stderr:dumped pgs 2020-03-11T22:55:09.228 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:77: _common_test: newprimary=4 2020-03-11T22:55:09.228 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:78: _common_test: kill_daemons 2020-03-11T22:55:09.229 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:55:09.229 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:55:09.229 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:55:09.229 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:55:09.229 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:55:09.540 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:55:09.540 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:80: _common_test: ERRORS=0 2020-03-11T22:55:09.540 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:81: _common_test: _objectstore_tool_nodown td/osd-backfill-recovery-log 4 --no-mon-config --pgid 1.0 --op log 2020-03-11T22:55:09.540 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/osd-backfill-recovery-log 2020-03-11T22:55:09.541 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:81: _common_test: tee td/osd-backfill-recovery-log/result.log 2020-03-11T22:55:09.541 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-03-11T22:55:09.541 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=4 2020-03-11T22:55:09.541 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-03-11T22:55:09.541 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/osd-backfill-recovery-log/4 2020-03-11T22:55:09.541 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-backfill-recovery-log/4 --no-mon-config --pgid 1.0 --op log 2020-03-11T22:55:10.135 INFO:tasks.workunit.client.0.smithi059.stdout:{ 2020-03-11T22:55:10.135 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_log_t": { 2020-03-11T22:55:10.136 INFO:tasks.workunit.client.0.smithi059.stdout: "head": "34'40", 2020-03-11T22:55:10.136 INFO:tasks.workunit.client.0.smithi059.stdout: "tail": "0'0", 2020-03-11T22:55:10.136 INFO:tasks.workunit.client.0.smithi059.stdout: "log": [ 2020-03-11T22:55:10.136 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:55:10.136 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "modify", 2020-03-11T22:55:10.136 INFO:tasks.workunit.client.0.smithi059.stdout: "object": "1:c00128d9:::obj-1:head", 2020-03-11T22:55:10.137 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'1", 2020-03-11T22:55:10.137 INFO:tasks.workunit.client.0.smithi059.stdout: "prior_version": "0'0", 2020-03-11T22:55:10.137 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4189.0:1", 2020-03-11T22:55:10.137 INFO:tasks.workunit.client.0.smithi059.stdout: "extra_reqids": [], 2020-03-11T22:55:10.137 INFO:tasks.workunit.client.0.smithi059.stdout: "mtime": "2020-03-11T22:54:50.011725+0000", 2020-03-11T22:55:10.137 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": 0, 2020-03-11T22:55:10.138 INFO:tasks.workunit.client.0.smithi059.stdout: "mod_desc": { 2020-03-11T22:55:10.138 INFO:tasks.workunit.client.0.smithi059.stdout: "object_mod_desc": { 2020-03-11T22:55:10.138 INFO:tasks.workunit.client.0.smithi059.stdout: "can_local_rollback": false, 2020-03-11T22:55:10.138 INFO:tasks.workunit.client.0.smithi059.stdout: "rollback_info_completed": false, 2020-03-11T22:55:10.138 INFO:tasks.workunit.client.0.smithi059.stdout: "ops": [] 2020-03-11T22:55:10.138 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.138 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.139 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_regions": { 2020-03-11T22:55:10.139 INFO:tasks.workunit.client.0.smithi059.stdout: "object_clean_regions": { 2020-03-11T22:55:10.139 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_offsets": "[6900~18446744073709544715]", 2020-03-11T22:55:10.139 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_omap": true, 2020-03-11T22:55:10.139 INFO:tasks.workunit.client.0.smithi059.stdout: "new_object": false 2020-03-11T22:55:10.139 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.139 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.139 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.140 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:55:10.140 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "modify", 2020-03-11T22:55:10.140 INFO:tasks.workunit.client.0.smithi059.stdout: "object": "1:d7942fc0:::obj-2:head", 2020-03-11T22:55:10.140 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'2", 2020-03-11T22:55:10.140 INFO:tasks.workunit.client.0.smithi059.stdout: "prior_version": "0'0", 2020-03-11T22:55:10.140 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4192.0:1", 2020-03-11T22:55:10.140 INFO:tasks.workunit.client.0.smithi059.stdout: "extra_reqids": [], 2020-03-11T22:55:10.141 INFO:tasks.workunit.client.0.smithi059.stdout: "mtime": "2020-03-11T22:54:50.061599+0000", 2020-03-11T22:55:10.141 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": 0, 2020-03-11T22:55:10.141 INFO:tasks.workunit.client.0.smithi059.stdout: "mod_desc": { 2020-03-11T22:55:10.141 INFO:tasks.workunit.client.0.smithi059.stdout: "object_mod_desc": { 2020-03-11T22:55:10.141 INFO:tasks.workunit.client.0.smithi059.stdout: "can_local_rollback": false, 2020-03-11T22:55:10.141 INFO:tasks.workunit.client.0.smithi059.stdout: "rollback_info_completed": false, 2020-03-11T22:55:10.142 INFO:tasks.workunit.client.0.smithi059.stdout: "ops": [] 2020-03-11T22:55:10.142 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.142 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.142 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_regions": { 2020-03-11T22:55:10.142 INFO:tasks.workunit.client.0.smithi059.stdout: "object_clean_regions": { 2020-03-11T22:55:10.142 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_offsets": "[6900~18446744073709544715]", 2020-03-11T22:55:10.142 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_omap": true, 2020-03-11T22:55:10.143 INFO:tasks.workunit.client.0.smithi059.stdout: "new_object": false 2020-03-11T22:55:10.143 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.143 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.143 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.143 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:55:10.143 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "modify", 2020-03-11T22:55:10.144 INFO:tasks.workunit.client.0.smithi059.stdout: "object": "1:1dda57ab:::obj-3:head", 2020-03-11T22:55:10.144 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'3", 2020-03-11T22:55:10.144 INFO:tasks.workunit.client.0.smithi059.stdout: "prior_version": "0'0", 2020-03-11T22:55:10.144 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4195.0:1", 2020-03-11T22:55:10.144 INFO:tasks.workunit.client.0.smithi059.stdout: "extra_reqids": [], 2020-03-11T22:55:10.145 INFO:tasks.workunit.client.0.smithi059.stdout: "mtime": "2020-03-11T22:54:50.127360+0000", 2020-03-11T22:55:10.145 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": 0, 2020-03-11T22:55:10.145 INFO:tasks.workunit.client.0.smithi059.stdout: "mod_desc": { 2020-03-11T22:55:10.145 INFO:tasks.workunit.client.0.smithi059.stdout: "object_mod_desc": { 2020-03-11T22:55:10.145 INFO:tasks.workunit.client.0.smithi059.stdout: "can_local_rollback": false, 2020-03-11T22:55:10.145 INFO:tasks.workunit.client.0.smithi059.stdout: "rollback_info_completed": false, 2020-03-11T22:55:10.145 INFO:tasks.workunit.client.0.smithi059.stdout: "ops": [] 2020-03-11T22:55:10.146 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.146 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.146 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_regions": { 2020-03-11T22:55:10.146 INFO:tasks.workunit.client.0.smithi059.stdout: "object_clean_regions": { 2020-03-11T22:55:10.146 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_offsets": "[6900~18446744073709544715]", 2020-03-11T22:55:10.147 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_omap": true, 2020-03-11T22:55:10.147 INFO:tasks.workunit.client.0.smithi059.stdout: "new_object": false 2020-03-11T22:55:10.147 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.147 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.147 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.148 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:55:10.148 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "modify", 2020-03-11T22:55:10.148 INFO:tasks.workunit.client.0.smithi059.stdout: "object": "1:ba1802d0:::obj-4:head", 2020-03-11T22:55:10.148 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'4", 2020-03-11T22:55:10.149 INFO:tasks.workunit.client.0.smithi059.stdout: "prior_version": "0'0", 2020-03-11T22:55:10.149 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4198.0:1", 2020-03-11T22:55:10.149 INFO:tasks.workunit.client.0.smithi059.stdout: "extra_reqids": [], 2020-03-11T22:55:10.149 INFO:tasks.workunit.client.0.smithi059.stdout: "mtime": "2020-03-11T22:54:50.208297+0000", 2020-03-11T22:55:10.150 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": 0, 2020-03-11T22:55:10.150 INFO:tasks.workunit.client.0.smithi059.stdout: "mod_desc": { 2020-03-11T22:55:10.150 INFO:tasks.workunit.client.0.smithi059.stdout: "object_mod_desc": { 2020-03-11T22:55:10.150 INFO:tasks.workunit.client.0.smithi059.stdout: "can_local_rollback": false, 2020-03-11T22:55:10.151 INFO:tasks.workunit.client.0.smithi059.stdout: "rollback_info_completed": false, 2020-03-11T22:55:10.151 INFO:tasks.workunit.client.0.smithi059.stdout: "ops": [] 2020-03-11T22:55:10.151 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.151 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.152 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_regions": { 2020-03-11T22:55:10.152 INFO:tasks.workunit.client.0.smithi059.stdout: "object_clean_regions": { 2020-03-11T22:55:10.152 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_offsets": "[6900~18446744073709544715]", 2020-03-11T22:55:10.153 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_omap": true, 2020-03-11T22:55:10.153 INFO:tasks.workunit.client.0.smithi059.stdout: "new_object": false 2020-03-11T22:55:10.153 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.153 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.154 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.154 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:55:10.154 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "modify", 2020-03-11T22:55:10.154 INFO:tasks.workunit.client.0.smithi059.stdout: "object": "1:d5d4a582:::obj-5:head", 2020-03-11T22:55:10.155 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'5", 2020-03-11T22:55:10.155 INFO:tasks.workunit.client.0.smithi059.stdout: "prior_version": "0'0", 2020-03-11T22:55:10.155 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4201.0:1", 2020-03-11T22:55:10.155 INFO:tasks.workunit.client.0.smithi059.stdout: "extra_reqids": [], 2020-03-11T22:55:10.156 INFO:tasks.workunit.client.0.smithi059.stdout: "mtime": "2020-03-11T22:54:50.255849+0000", 2020-03-11T22:55:10.156 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": 0, 2020-03-11T22:55:10.156 INFO:tasks.workunit.client.0.smithi059.stdout: "mod_desc": { 2020-03-11T22:55:10.156 INFO:tasks.workunit.client.0.smithi059.stdout: "object_mod_desc": { 2020-03-11T22:55:10.156 INFO:tasks.workunit.client.0.smithi059.stdout: "can_local_rollback": false, 2020-03-11T22:55:10.157 INFO:tasks.workunit.client.0.smithi059.stdout: "rollback_info_completed": false, 2020-03-11T22:55:10.157 INFO:tasks.workunit.client.0.smithi059.stdout: "ops": [] 2020-03-11T22:55:10.157 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.157 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.158 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_regions": { 2020-03-11T22:55:10.158 INFO:tasks.workunit.client.0.smithi059.stdout: "object_clean_regions": { 2020-03-11T22:55:10.158 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_offsets": "[6900~18446744073709544715]", 2020-03-11T22:55:10.158 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_omap": true, 2020-03-11T22:55:10.159 INFO:tasks.workunit.client.0.smithi059.stdout: "new_object": false 2020-03-11T22:55:10.159 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.159 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.159 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.159 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:55:10.160 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "modify", 2020-03-11T22:55:10.160 INFO:tasks.workunit.client.0.smithi059.stdout: "object": "1:25cac09a:::obj-6:head", 2020-03-11T22:55:10.160 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'6", 2020-03-11T22:55:10.160 INFO:tasks.workunit.client.0.smithi059.stdout: "prior_version": "0'0", 2020-03-11T22:55:10.160 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4204.0:1", 2020-03-11T22:55:10.161 INFO:tasks.workunit.client.0.smithi059.stdout: "extra_reqids": [], 2020-03-11T22:55:10.161 INFO:tasks.workunit.client.0.smithi059.stdout: "mtime": "2020-03-11T22:54:50.314815+0000", 2020-03-11T22:55:10.161 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": 0, 2020-03-11T22:55:10.161 INFO:tasks.workunit.client.0.smithi059.stdout: "mod_desc": { 2020-03-11T22:55:10.162 INFO:tasks.workunit.client.0.smithi059.stdout: "object_mod_desc": { 2020-03-11T22:55:10.162 INFO:tasks.workunit.client.0.smithi059.stdout: "can_local_rollback": false, 2020-03-11T22:55:10.162 INFO:tasks.workunit.client.0.smithi059.stdout: "rollback_info_completed": false, 2020-03-11T22:55:10.162 INFO:tasks.workunit.client.0.smithi059.stdout: "ops": [] 2020-03-11T22:55:10.163 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.163 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.163 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_regions": { 2020-03-11T22:55:10.163 INFO:tasks.workunit.client.0.smithi059.stdout: "object_clean_regions": { 2020-03-11T22:55:10.164 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_offsets": "[6900~18446744073709544715]", 2020-03-11T22:55:10.164 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_omap": true, 2020-03-11T22:55:10.164 INFO:tasks.workunit.client.0.smithi059.stdout: "new_object": false 2020-03-11T22:55:10.164 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.164 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.165 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.165 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:55:10.165 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "modify", 2020-03-11T22:55:10.165 INFO:tasks.workunit.client.0.smithi059.stdout: "object": "1:b11b7838:::obj-7:head", 2020-03-11T22:55:10.166 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'7", 2020-03-11T22:55:10.166 INFO:tasks.workunit.client.0.smithi059.stdout: "prior_version": "0'0", 2020-03-11T22:55:10.166 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4207.0:1", 2020-03-11T22:55:10.166 INFO:tasks.workunit.client.0.smithi059.stdout: "extra_reqids": [], 2020-03-11T22:55:10.166 INFO:tasks.workunit.client.0.smithi059.stdout: "mtime": "2020-03-11T22:54:50.351063+0000", 2020-03-11T22:55:10.167 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": 0, 2020-03-11T22:55:10.167 INFO:tasks.workunit.client.0.smithi059.stdout: "mod_desc": { 2020-03-11T22:55:10.167 INFO:tasks.workunit.client.0.smithi059.stdout: "object_mod_desc": { 2020-03-11T22:55:10.167 INFO:tasks.workunit.client.0.smithi059.stdout: "can_local_rollback": false, 2020-03-11T22:55:10.167 INFO:tasks.workunit.client.0.smithi059.stdout: "rollback_info_completed": false, 2020-03-11T22:55:10.168 INFO:tasks.workunit.client.0.smithi059.stdout: "ops": [] 2020-03-11T22:55:10.168 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.168 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.168 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_regions": { 2020-03-11T22:55:10.169 INFO:tasks.workunit.client.0.smithi059.stdout: "object_clean_regions": { 2020-03-11T22:55:10.169 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_offsets": "[6900~18446744073709544715]", 2020-03-11T22:55:10.169 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_omap": true, 2020-03-11T22:55:10.169 INFO:tasks.workunit.client.0.smithi059.stdout: "new_object": false 2020-03-11T22:55:10.169 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.170 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.170 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.170 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:55:10.170 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "modify", 2020-03-11T22:55:10.171 INFO:tasks.workunit.client.0.smithi059.stdout: "object": "1:a0f702de:::obj-8:head", 2020-03-11T22:55:10.171 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'8", 2020-03-11T22:55:10.171 INFO:tasks.workunit.client.0.smithi059.stdout: "prior_version": "0'0", 2020-03-11T22:55:10.171 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4210.0:1", 2020-03-11T22:55:10.171 INFO:tasks.workunit.client.0.smithi059.stdout: "extra_reqids": [], 2020-03-11T22:55:10.172 INFO:tasks.workunit.client.0.smithi059.stdout: "mtime": "2020-03-11T22:54:50.424149+0000", 2020-03-11T22:55:10.172 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": 0, 2020-03-11T22:55:10.172 INFO:tasks.workunit.client.0.smithi059.stdout: "mod_desc": { 2020-03-11T22:55:10.172 INFO:tasks.workunit.client.0.smithi059.stdout: "object_mod_desc": { 2020-03-11T22:55:10.172 INFO:tasks.workunit.client.0.smithi059.stdout: "can_local_rollback": false, 2020-03-11T22:55:10.173 INFO:tasks.workunit.client.0.smithi059.stdout: "rollback_info_completed": false, 2020-03-11T22:55:10.173 INFO:tasks.workunit.client.0.smithi059.stdout: "ops": [] 2020-03-11T22:55:10.173 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.173 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.174 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_regions": { 2020-03-11T22:55:10.174 INFO:tasks.workunit.client.0.smithi059.stdout: "object_clean_regions": { 2020-03-11T22:55:10.174 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_offsets": "[6900~18446744073709544715]", 2020-03-11T22:55:10.174 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_omap": true, 2020-03-11T22:55:10.174 INFO:tasks.workunit.client.0.smithi059.stdout: "new_object": false 2020-03-11T22:55:10.174 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.174 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.175 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.175 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:55:10.175 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "modify", 2020-03-11T22:55:10.175 INFO:tasks.workunit.client.0.smithi059.stdout: "object": "1:1e5c3677:::obj-9:head", 2020-03-11T22:55:10.175 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'9", 2020-03-11T22:55:10.175 INFO:tasks.workunit.client.0.smithi059.stdout: "prior_version": "0'0", 2020-03-11T22:55:10.175 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4213.0:1", 2020-03-11T22:55:10.175 INFO:tasks.workunit.client.0.smithi059.stdout: "extra_reqids": [], 2020-03-11T22:55:10.176 INFO:tasks.workunit.client.0.smithi059.stdout: "mtime": "2020-03-11T22:54:50.524615+0000", 2020-03-11T22:55:10.176 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": 0, 2020-03-11T22:55:10.176 INFO:tasks.workunit.client.0.smithi059.stdout: "mod_desc": { 2020-03-11T22:55:10.176 INFO:tasks.workunit.client.0.smithi059.stdout: "object_mod_desc": { 2020-03-11T22:55:10.176 INFO:tasks.workunit.client.0.smithi059.stdout: "can_local_rollback": false, 2020-03-11T22:55:10.176 INFO:tasks.workunit.client.0.smithi059.stdout: "rollback_info_completed": false, 2020-03-11T22:55:10.176 INFO:tasks.workunit.client.0.smithi059.stdout: "ops": [] 2020-03-11T22:55:10.177 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.177 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.177 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_regions": { 2020-03-11T22:55:10.177 INFO:tasks.workunit.client.0.smithi059.stdout: "object_clean_regions": { 2020-03-11T22:55:10.177 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_offsets": "[6900~18446744073709544715]", 2020-03-11T22:55:10.177 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_omap": true, 2020-03-11T22:55:10.177 INFO:tasks.workunit.client.0.smithi059.stdout: "new_object": false 2020-03-11T22:55:10.178 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.178 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.178 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.178 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:55:10.178 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "modify", 2020-03-11T22:55:10.178 INFO:tasks.workunit.client.0.smithi059.stdout: "object": "1:86dc045c:::obj-10:head", 2020-03-11T22:55:10.178 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'10", 2020-03-11T22:55:10.179 INFO:tasks.workunit.client.0.smithi059.stdout: "prior_version": "0'0", 2020-03-11T22:55:10.179 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4216.0:1", 2020-03-11T22:55:10.179 INFO:tasks.workunit.client.0.smithi059.stdout: "extra_reqids": [], 2020-03-11T22:55:10.179 INFO:tasks.workunit.client.0.smithi059.stdout: "mtime": "2020-03-11T22:54:50.594282+0000", 2020-03-11T22:55:10.179 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": 0, 2020-03-11T22:55:10.179 INFO:tasks.workunit.client.0.smithi059.stdout: "mod_desc": { 2020-03-11T22:55:10.179 INFO:tasks.workunit.client.0.smithi059.stdout: "object_mod_desc": { 2020-03-11T22:55:10.180 INFO:tasks.workunit.client.0.smithi059.stdout: "can_local_rollback": false, 2020-03-11T22:55:10.180 INFO:tasks.workunit.client.0.smithi059.stdout: "rollback_info_completed": false, 2020-03-11T22:55:10.180 INFO:tasks.workunit.client.0.smithi059.stdout: "ops": [] 2020-03-11T22:55:10.180 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.180 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.180 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_regions": { 2020-03-11T22:55:10.180 INFO:tasks.workunit.client.0.smithi059.stdout: "object_clean_regions": { 2020-03-11T22:55:10.181 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_offsets": "[6900~18446744073709544715]", 2020-03-11T22:55:10.181 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_omap": true, 2020-03-11T22:55:10.181 INFO:tasks.workunit.client.0.smithi059.stdout: "new_object": false 2020-03-11T22:55:10.181 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.181 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.181 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.181 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:55:10.182 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "modify", 2020-03-11T22:55:10.182 INFO:tasks.workunit.client.0.smithi059.stdout: "object": "1:1e6eca61:::obj-11:head", 2020-03-11T22:55:10.182 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'11", 2020-03-11T22:55:10.182 INFO:tasks.workunit.client.0.smithi059.stdout: "prior_version": "0'0", 2020-03-11T22:55:10.182 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4219.0:1", 2020-03-11T22:55:10.182 INFO:tasks.workunit.client.0.smithi059.stdout: "extra_reqids": [], 2020-03-11T22:55:10.182 INFO:tasks.workunit.client.0.smithi059.stdout: "mtime": "2020-03-11T22:54:50.674860+0000", 2020-03-11T22:55:10.183 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": 0, 2020-03-11T22:55:10.183 INFO:tasks.workunit.client.0.smithi059.stdout: "mod_desc": { 2020-03-11T22:55:10.183 INFO:tasks.workunit.client.0.smithi059.stdout: "object_mod_desc": { 2020-03-11T22:55:10.183 INFO:tasks.workunit.client.0.smithi059.stdout: "can_local_rollback": false, 2020-03-11T22:55:10.183 INFO:tasks.workunit.client.0.smithi059.stdout: "rollback_info_completed": false, 2020-03-11T22:55:10.183 INFO:tasks.workunit.client.0.smithi059.stdout: "ops": [] 2020-03-11T22:55:10.183 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.183 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.184 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_regions": { 2020-03-11T22:55:10.184 INFO:tasks.workunit.client.0.smithi059.stdout: "object_clean_regions": { 2020-03-11T22:55:10.184 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_offsets": "[6900~18446744073709544715]", 2020-03-11T22:55:10.184 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_omap": true, 2020-03-11T22:55:10.184 INFO:tasks.workunit.client.0.smithi059.stdout: "new_object": false 2020-03-11T22:55:10.184 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.184 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.185 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.185 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:55:10.185 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "modify", 2020-03-11T22:55:10.185 INFO:tasks.workunit.client.0.smithi059.stdout: "object": "1:18dee445:::obj-12:head", 2020-03-11T22:55:10.185 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'12", 2020-03-11T22:55:10.185 INFO:tasks.workunit.client.0.smithi059.stdout: "prior_version": "0'0", 2020-03-11T22:55:10.185 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4222.0:1", 2020-03-11T22:55:10.186 INFO:tasks.workunit.client.0.smithi059.stdout: "extra_reqids": [], 2020-03-11T22:55:10.186 INFO:tasks.workunit.client.0.smithi059.stdout: "mtime": "2020-03-11T22:54:50.729299+0000", 2020-03-11T22:55:10.186 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": 0, 2020-03-11T22:55:10.186 INFO:tasks.workunit.client.0.smithi059.stdout: "mod_desc": { 2020-03-11T22:55:10.186 INFO:tasks.workunit.client.0.smithi059.stdout: "object_mod_desc": { 2020-03-11T22:55:10.186 INFO:tasks.workunit.client.0.smithi059.stdout: "can_local_rollback": false, 2020-03-11T22:55:10.186 INFO:tasks.workunit.client.0.smithi059.stdout: "rollback_info_completed": false, 2020-03-11T22:55:10.187 INFO:tasks.workunit.client.0.smithi059.stdout: "ops": [] 2020-03-11T22:55:10.187 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.187 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.187 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_regions": { 2020-03-11T22:55:10.187 INFO:tasks.workunit.client.0.smithi059.stdout: "object_clean_regions": { 2020-03-11T22:55:10.187 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_offsets": "[6900~18446744073709544715]", 2020-03-11T22:55:10.187 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_omap": true, 2020-03-11T22:55:10.188 INFO:tasks.workunit.client.0.smithi059.stdout: "new_object": false 2020-03-11T22:55:10.188 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.188 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.188 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.188 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:55:10.188 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "modify", 2020-03-11T22:55:10.189 INFO:tasks.workunit.client.0.smithi059.stdout: "object": "1:283c5c90:::obj-13:head", 2020-03-11T22:55:10.189 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'13", 2020-03-11T22:55:10.189 INFO:tasks.workunit.client.0.smithi059.stdout: "prior_version": "0'0", 2020-03-11T22:55:10.189 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4225.0:1", 2020-03-11T22:55:10.189 INFO:tasks.workunit.client.0.smithi059.stdout: "extra_reqids": [], 2020-03-11T22:55:10.189 INFO:tasks.workunit.client.0.smithi059.stdout: "mtime": "2020-03-11T22:54:50.786535+0000", 2020-03-11T22:55:10.189 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": 0, 2020-03-11T22:55:10.190 INFO:tasks.workunit.client.0.smithi059.stdout: "mod_desc": { 2020-03-11T22:55:10.190 INFO:tasks.workunit.client.0.smithi059.stdout: "object_mod_desc": { 2020-03-11T22:55:10.190 INFO:tasks.workunit.client.0.smithi059.stdout: "can_local_rollback": false, 2020-03-11T22:55:10.190 INFO:tasks.workunit.client.0.smithi059.stdout: "rollback_info_completed": false, 2020-03-11T22:55:10.190 INFO:tasks.workunit.client.0.smithi059.stdout: "ops": [] 2020-03-11T22:55:10.190 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.190 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.191 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_regions": { 2020-03-11T22:55:10.191 INFO:tasks.workunit.client.0.smithi059.stdout: "object_clean_regions": { 2020-03-11T22:55:10.191 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_offsets": "[6900~18446744073709544715]", 2020-03-11T22:55:10.191 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_omap": true, 2020-03-11T22:55:10.191 INFO:tasks.workunit.client.0.smithi059.stdout: "new_object": false 2020-03-11T22:55:10.191 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.191 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.192 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.192 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:55:10.192 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "modify", 2020-03-11T22:55:10.192 INFO:tasks.workunit.client.0.smithi059.stdout: "object": "1:cc896bf0:::obj-14:head", 2020-03-11T22:55:10.192 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'14", 2020-03-11T22:55:10.192 INFO:tasks.workunit.client.0.smithi059.stdout: "prior_version": "0'0", 2020-03-11T22:55:10.192 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4228.0:1", 2020-03-11T22:55:10.192 INFO:tasks.workunit.client.0.smithi059.stdout: "extra_reqids": [], 2020-03-11T22:55:10.193 INFO:tasks.workunit.client.0.smithi059.stdout: "mtime": "2020-03-11T22:54:50.821395+0000", 2020-03-11T22:55:10.193 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": 0, 2020-03-11T22:55:10.193 INFO:tasks.workunit.client.0.smithi059.stdout: "mod_desc": { 2020-03-11T22:55:10.193 INFO:tasks.workunit.client.0.smithi059.stdout: "object_mod_desc": { 2020-03-11T22:55:10.193 INFO:tasks.workunit.client.0.smithi059.stdout: "can_local_rollback": false, 2020-03-11T22:55:10.193 INFO:tasks.workunit.client.0.smithi059.stdout: "rollback_info_completed": false, 2020-03-11T22:55:10.193 INFO:tasks.workunit.client.0.smithi059.stdout: "ops": [] 2020-03-11T22:55:10.194 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.194 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.194 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_regions": { 2020-03-11T22:55:10.194 INFO:tasks.workunit.client.0.smithi059.stdout: "object_clean_regions": { 2020-03-11T22:55:10.194 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_offsets": "[6900~18446744073709544715]", 2020-03-11T22:55:10.194 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_omap": true, 2020-03-11T22:55:10.194 INFO:tasks.workunit.client.0.smithi059.stdout: "new_object": false 2020-03-11T22:55:10.195 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.195 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.195 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.195 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:55:10.195 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "modify", 2020-03-11T22:55:10.195 INFO:tasks.workunit.client.0.smithi059.stdout: "object": "1:a581f807:::obj-15:head", 2020-03-11T22:55:10.195 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'15", 2020-03-11T22:55:10.196 INFO:tasks.workunit.client.0.smithi059.stdout: "prior_version": "0'0", 2020-03-11T22:55:10.196 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4231.0:1", 2020-03-11T22:55:10.196 INFO:tasks.workunit.client.0.smithi059.stdout: "extra_reqids": [], 2020-03-11T22:55:10.196 INFO:tasks.workunit.client.0.smithi059.stdout: "mtime": "2020-03-11T22:54:50.925701+0000", 2020-03-11T22:55:10.196 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": 0, 2020-03-11T22:55:10.196 INFO:tasks.workunit.client.0.smithi059.stdout: "mod_desc": { 2020-03-11T22:55:10.197 INFO:tasks.workunit.client.0.smithi059.stdout: "object_mod_desc": { 2020-03-11T22:55:10.197 INFO:tasks.workunit.client.0.smithi059.stdout: "can_local_rollback": false, 2020-03-11T22:55:10.197 INFO:tasks.workunit.client.0.smithi059.stdout: "rollback_info_completed": false, 2020-03-11T22:55:10.197 INFO:tasks.workunit.client.0.smithi059.stdout: "ops": [] 2020-03-11T22:55:10.197 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.197 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.197 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_regions": { 2020-03-11T22:55:10.198 INFO:tasks.workunit.client.0.smithi059.stdout: "object_clean_regions": { 2020-03-11T22:55:10.198 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_offsets": "[6900~18446744073709544715]", 2020-03-11T22:55:10.198 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_omap": true, 2020-03-11T22:55:10.198 INFO:tasks.workunit.client.0.smithi059.stdout: "new_object": false 2020-03-11T22:55:10.198 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.198 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.198 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.199 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:55:10.199 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "modify", 2020-03-11T22:55:10.199 INFO:tasks.workunit.client.0.smithi059.stdout: "object": "1:c902ca7c:::obj-16:head", 2020-03-11T22:55:10.199 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'16", 2020-03-11T22:55:10.199 INFO:tasks.workunit.client.0.smithi059.stdout: "prior_version": "0'0", 2020-03-11T22:55:10.199 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4234.0:1", 2020-03-11T22:55:10.199 INFO:tasks.workunit.client.0.smithi059.stdout: "extra_reqids": [], 2020-03-11T22:55:10.200 INFO:tasks.workunit.client.0.smithi059.stdout: "mtime": "2020-03-11T22:54:51.026293+0000", 2020-03-11T22:55:10.200 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": 0, 2020-03-11T22:55:10.200 INFO:tasks.workunit.client.0.smithi059.stdout: "mod_desc": { 2020-03-11T22:55:10.200 INFO:tasks.workunit.client.0.smithi059.stdout: "object_mod_desc": { 2020-03-11T22:55:10.200 INFO:tasks.workunit.client.0.smithi059.stdout: "can_local_rollback": false, 2020-03-11T22:55:10.200 INFO:tasks.workunit.client.0.smithi059.stdout: "rollback_info_completed": false, 2020-03-11T22:55:10.200 INFO:tasks.workunit.client.0.smithi059.stdout: "ops": [] 2020-03-11T22:55:10.200 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.201 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.201 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_regions": { 2020-03-11T22:55:10.201 INFO:tasks.workunit.client.0.smithi059.stdout: "object_clean_regions": { 2020-03-11T22:55:10.201 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_offsets": "[6900~18446744073709544715]", 2020-03-11T22:55:10.201 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_omap": true, 2020-03-11T22:55:10.201 INFO:tasks.workunit.client.0.smithi059.stdout: "new_object": false 2020-03-11T22:55:10.201 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.202 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.202 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.202 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:55:10.202 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "modify", 2020-03-11T22:55:10.202 INFO:tasks.workunit.client.0.smithi059.stdout: "object": "1:655283c8:::obj-17:head", 2020-03-11T22:55:10.202 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'17", 2020-03-11T22:55:10.202 INFO:tasks.workunit.client.0.smithi059.stdout: "prior_version": "0'0", 2020-03-11T22:55:10.203 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4237.0:1", 2020-03-11T22:55:10.203 INFO:tasks.workunit.client.0.smithi059.stdout: "extra_reqids": [], 2020-03-11T22:55:10.203 INFO:tasks.workunit.client.0.smithi059.stdout: "mtime": "2020-03-11T22:54:51.063672+0000", 2020-03-11T22:55:10.203 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": 0, 2020-03-11T22:55:10.203 INFO:tasks.workunit.client.0.smithi059.stdout: "mod_desc": { 2020-03-11T22:55:10.203 INFO:tasks.workunit.client.0.smithi059.stdout: "object_mod_desc": { 2020-03-11T22:55:10.203 INFO:tasks.workunit.client.0.smithi059.stdout: "can_local_rollback": false, 2020-03-11T22:55:10.204 INFO:tasks.workunit.client.0.smithi059.stdout: "rollback_info_completed": false, 2020-03-11T22:55:10.204 INFO:tasks.workunit.client.0.smithi059.stdout: "ops": [] 2020-03-11T22:55:10.204 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.204 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.204 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_regions": { 2020-03-11T22:55:10.204 INFO:tasks.workunit.client.0.smithi059.stdout: "object_clean_regions": { 2020-03-11T22:55:10.204 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_offsets": "[6900~18446744073709544715]", 2020-03-11T22:55:10.205 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_omap": true, 2020-03-11T22:55:10.205 INFO:tasks.workunit.client.0.smithi059.stdout: "new_object": false 2020-03-11T22:55:10.205 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.205 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.205 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.205 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:55:10.205 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "modify", 2020-03-11T22:55:10.206 INFO:tasks.workunit.client.0.smithi059.stdout: "object": "1:d759d762:::obj-18:head", 2020-03-11T22:55:10.206 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'18", 2020-03-11T22:55:10.206 INFO:tasks.workunit.client.0.smithi059.stdout: "prior_version": "0'0", 2020-03-11T22:55:10.206 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4240.0:1", 2020-03-11T22:55:10.206 INFO:tasks.workunit.client.0.smithi059.stdout: "extra_reqids": [], 2020-03-11T22:55:10.206 INFO:tasks.workunit.client.0.smithi059.stdout: "mtime": "2020-03-11T22:54:51.155516+0000", 2020-03-11T22:55:10.206 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": 0, 2020-03-11T22:55:10.207 INFO:tasks.workunit.client.0.smithi059.stdout: "mod_desc": { 2020-03-11T22:55:10.207 INFO:tasks.workunit.client.0.smithi059.stdout: "object_mod_desc": { 2020-03-11T22:55:10.207 INFO:tasks.workunit.client.0.smithi059.stdout: "can_local_rollback": false, 2020-03-11T22:55:10.207 INFO:tasks.workunit.client.0.smithi059.stdout: "rollback_info_completed": false, 2020-03-11T22:55:10.207 INFO:tasks.workunit.client.0.smithi059.stdout: "ops": [] 2020-03-11T22:55:10.207 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.207 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.208 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_regions": { 2020-03-11T22:55:10.208 INFO:tasks.workunit.client.0.smithi059.stdout: "object_clean_regions": { 2020-03-11T22:55:10.208 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_offsets": "[6900~18446744073709544715]", 2020-03-11T22:55:10.208 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_omap": true, 2020-03-11T22:55:10.208 INFO:tasks.workunit.client.0.smithi059.stdout: "new_object": false 2020-03-11T22:55:10.208 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.208 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.208 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.209 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:55:10.209 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "modify", 2020-03-11T22:55:10.209 INFO:tasks.workunit.client.0.smithi059.stdout: "object": "1:1b934a2d:::obj-19:head", 2020-03-11T22:55:10.209 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'19", 2020-03-11T22:55:10.209 INFO:tasks.workunit.client.0.smithi059.stdout: "prior_version": "0'0", 2020-03-11T22:55:10.209 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4243.0:1", 2020-03-11T22:55:10.209 INFO:tasks.workunit.client.0.smithi059.stdout: "extra_reqids": [], 2020-03-11T22:55:10.210 INFO:tasks.workunit.client.0.smithi059.stdout: "mtime": "2020-03-11T22:54:51.260250+0000", 2020-03-11T22:55:10.210 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": 0, 2020-03-11T22:55:10.210 INFO:tasks.workunit.client.0.smithi059.stdout: "mod_desc": { 2020-03-11T22:55:10.210 INFO:tasks.workunit.client.0.smithi059.stdout: "object_mod_desc": { 2020-03-11T22:55:10.210 INFO:tasks.workunit.client.0.smithi059.stdout: "can_local_rollback": false, 2020-03-11T22:55:10.210 INFO:tasks.workunit.client.0.smithi059.stdout: "rollback_info_completed": false, 2020-03-11T22:55:10.210 INFO:tasks.workunit.client.0.smithi059.stdout: "ops": [] 2020-03-11T22:55:10.211 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.211 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.211 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_regions": { 2020-03-11T22:55:10.211 INFO:tasks.workunit.client.0.smithi059.stdout: "object_clean_regions": { 2020-03-11T22:55:10.211 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_offsets": "[6900~18446744073709544715]", 2020-03-11T22:55:10.211 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_omap": true, 2020-03-11T22:55:10.211 INFO:tasks.workunit.client.0.smithi059.stdout: "new_object": false 2020-03-11T22:55:10.212 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.212 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.212 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.212 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:55:10.212 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "modify", 2020-03-11T22:55:10.212 INFO:tasks.workunit.client.0.smithi059.stdout: "object": "1:1d0ea2ca:::obj-20:head", 2020-03-11T22:55:10.212 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'20", 2020-03-11T22:55:10.213 INFO:tasks.workunit.client.0.smithi059.stdout: "prior_version": "0'0", 2020-03-11T22:55:10.213 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4246.0:1", 2020-03-11T22:55:10.213 INFO:tasks.workunit.client.0.smithi059.stdout: "extra_reqids": [], 2020-03-11T22:55:10.213 INFO:tasks.workunit.client.0.smithi059.stdout: "mtime": "2020-03-11T22:54:51.304881+0000", 2020-03-11T22:55:10.213 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": 0, 2020-03-11T22:55:10.213 INFO:tasks.workunit.client.0.smithi059.stdout: "mod_desc": { 2020-03-11T22:55:10.213 INFO:tasks.workunit.client.0.smithi059.stdout: "object_mod_desc": { 2020-03-11T22:55:10.214 INFO:tasks.workunit.client.0.smithi059.stdout: "can_local_rollback": false, 2020-03-11T22:55:10.214 INFO:tasks.workunit.client.0.smithi059.stdout: "rollback_info_completed": false, 2020-03-11T22:55:10.214 INFO:tasks.workunit.client.0.smithi059.stdout: "ops": [] 2020-03-11T22:55:10.214 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.214 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.214 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_regions": { 2020-03-11T22:55:10.214 INFO:tasks.workunit.client.0.smithi059.stdout: "object_clean_regions": { 2020-03-11T22:55:10.215 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_offsets": "[6900~18446744073709544715]", 2020-03-11T22:55:10.215 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_omap": true, 2020-03-11T22:55:10.215 INFO:tasks.workunit.client.0.smithi059.stdout: "new_object": false 2020-03-11T22:55:10.215 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.215 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.215 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.215 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:55:10.216 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "modify", 2020-03-11T22:55:10.216 INFO:tasks.workunit.client.0.smithi059.stdout: "object": "1:ab396690:::obj-21:head", 2020-03-11T22:55:10.216 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'21", 2020-03-11T22:55:10.216 INFO:tasks.workunit.client.0.smithi059.stdout: "prior_version": "0'0", 2020-03-11T22:55:10.216 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4249.0:1", 2020-03-11T22:55:10.216 INFO:tasks.workunit.client.0.smithi059.stdout: "extra_reqids": [], 2020-03-11T22:55:10.216 INFO:tasks.workunit.client.0.smithi059.stdout: "mtime": "2020-03-11T22:54:51.338706+0000", 2020-03-11T22:55:10.217 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": 0, 2020-03-11T22:55:10.217 INFO:tasks.workunit.client.0.smithi059.stdout: "mod_desc": { 2020-03-11T22:55:10.217 INFO:tasks.workunit.client.0.smithi059.stdout: "object_mod_desc": { 2020-03-11T22:55:10.217 INFO:tasks.workunit.client.0.smithi059.stdout: "can_local_rollback": false, 2020-03-11T22:55:10.217 INFO:tasks.workunit.client.0.smithi059.stdout: "rollback_info_completed": false, 2020-03-11T22:55:10.217 INFO:tasks.workunit.client.0.smithi059.stdout: "ops": [] 2020-03-11T22:55:10.217 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.217 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.218 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_regions": { 2020-03-11T22:55:10.218 INFO:tasks.workunit.client.0.smithi059.stdout: "object_clean_regions": { 2020-03-11T22:55:10.218 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_offsets": "[6900~18446744073709544715]", 2020-03-11T22:55:10.218 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_omap": true, 2020-03-11T22:55:10.218 INFO:tasks.workunit.client.0.smithi059.stdout: "new_object": false 2020-03-11T22:55:10.218 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.218 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.219 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.219 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:55:10.219 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "modify", 2020-03-11T22:55:10.219 INFO:tasks.workunit.client.0.smithi059.stdout: "object": "1:254983db:::obj-22:head", 2020-03-11T22:55:10.219 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'22", 2020-03-11T22:55:10.219 INFO:tasks.workunit.client.0.smithi059.stdout: "prior_version": "0'0", 2020-03-11T22:55:10.219 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4252.0:1", 2020-03-11T22:55:10.220 INFO:tasks.workunit.client.0.smithi059.stdout: "extra_reqids": [], 2020-03-11T22:55:10.220 INFO:tasks.workunit.client.0.smithi059.stdout: "mtime": "2020-03-11T22:54:51.477397+0000", 2020-03-11T22:55:10.220 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": 0, 2020-03-11T22:55:10.220 INFO:tasks.workunit.client.0.smithi059.stdout: "mod_desc": { 2020-03-11T22:55:10.220 INFO:tasks.workunit.client.0.smithi059.stdout: "object_mod_desc": { 2020-03-11T22:55:10.220 INFO:tasks.workunit.client.0.smithi059.stdout: "can_local_rollback": false, 2020-03-11T22:55:10.220 INFO:tasks.workunit.client.0.smithi059.stdout: "rollback_info_completed": false, 2020-03-11T22:55:10.221 INFO:tasks.workunit.client.0.smithi059.stdout: "ops": [] 2020-03-11T22:55:10.221 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.221 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.221 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_regions": { 2020-03-11T22:55:10.221 INFO:tasks.workunit.client.0.smithi059.stdout: "object_clean_regions": { 2020-03-11T22:55:10.221 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_offsets": "[6900~18446744073709544715]", 2020-03-11T22:55:10.221 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_omap": true, 2020-03-11T22:55:10.222 INFO:tasks.workunit.client.0.smithi059.stdout: "new_object": false 2020-03-11T22:55:10.222 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.222 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.222 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.222 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:55:10.222 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "modify", 2020-03-11T22:55:10.222 INFO:tasks.workunit.client.0.smithi059.stdout: "object": "1:e656299f:::obj-23:head", 2020-03-11T22:55:10.223 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'23", 2020-03-11T22:55:10.223 INFO:tasks.workunit.client.0.smithi059.stdout: "prior_version": "0'0", 2020-03-11T22:55:10.223 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4255.0:1", 2020-03-11T22:55:10.223 INFO:tasks.workunit.client.0.smithi059.stdout: "extra_reqids": [], 2020-03-11T22:55:10.223 INFO:tasks.workunit.client.0.smithi059.stdout: "mtime": "2020-03-11T22:54:51.552628+0000", 2020-03-11T22:55:10.223 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": 0, 2020-03-11T22:55:10.223 INFO:tasks.workunit.client.0.smithi059.stdout: "mod_desc": { 2020-03-11T22:55:10.223 INFO:tasks.workunit.client.0.smithi059.stdout: "object_mod_desc": { 2020-03-11T22:55:10.224 INFO:tasks.workunit.client.0.smithi059.stdout: "can_local_rollback": false, 2020-03-11T22:55:10.224 INFO:tasks.workunit.client.0.smithi059.stdout: "rollback_info_completed": false, 2020-03-11T22:55:10.224 INFO:tasks.workunit.client.0.smithi059.stdout: "ops": [] 2020-03-11T22:55:10.224 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.224 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.224 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_regions": { 2020-03-11T22:55:10.224 INFO:tasks.workunit.client.0.smithi059.stdout: "object_clean_regions": { 2020-03-11T22:55:10.225 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_offsets": "[6900~18446744073709544715]", 2020-03-11T22:55:10.225 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_omap": true, 2020-03-11T22:55:10.225 INFO:tasks.workunit.client.0.smithi059.stdout: "new_object": false 2020-03-11T22:55:10.225 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.225 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.225 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.225 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:55:10.226 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "modify", 2020-03-11T22:55:10.226 INFO:tasks.workunit.client.0.smithi059.stdout: "object": "1:5cdf412b:::obj-24:head", 2020-03-11T22:55:10.226 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'24", 2020-03-11T22:55:10.226 INFO:tasks.workunit.client.0.smithi059.stdout: "prior_version": "0'0", 2020-03-11T22:55:10.226 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4258.0:1", 2020-03-11T22:55:10.226 INFO:tasks.workunit.client.0.smithi059.stdout: "extra_reqids": [], 2020-03-11T22:55:10.226 INFO:tasks.workunit.client.0.smithi059.stdout: "mtime": "2020-03-11T22:54:51.588635+0000", 2020-03-11T22:55:10.227 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": 0, 2020-03-11T22:55:10.227 INFO:tasks.workunit.client.0.smithi059.stdout: "mod_desc": { 2020-03-11T22:55:10.227 INFO:tasks.workunit.client.0.smithi059.stdout: "object_mod_desc": { 2020-03-11T22:55:10.227 INFO:tasks.workunit.client.0.smithi059.stdout: "can_local_rollback": false, 2020-03-11T22:55:10.227 INFO:tasks.workunit.client.0.smithi059.stdout: "rollback_info_completed": false, 2020-03-11T22:55:10.227 INFO:tasks.workunit.client.0.smithi059.stdout: "ops": [] 2020-03-11T22:55:10.227 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.227 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.228 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_regions": { 2020-03-11T22:55:10.228 INFO:tasks.workunit.client.0.smithi059.stdout: "object_clean_regions": { 2020-03-11T22:55:10.228 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_offsets": "[6900~18446744073709544715]", 2020-03-11T22:55:10.228 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_omap": true, 2020-03-11T22:55:10.228 INFO:tasks.workunit.client.0.smithi059.stdout: "new_object": false 2020-03-11T22:55:10.228 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.228 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.229 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.229 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:55:10.229 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "modify", 2020-03-11T22:55:10.229 INFO:tasks.workunit.client.0.smithi059.stdout: "object": "1:cacae143:::obj-25:head", 2020-03-11T22:55:10.229 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'25", 2020-03-11T22:55:10.229 INFO:tasks.workunit.client.0.smithi059.stdout: "prior_version": "0'0", 2020-03-11T22:55:10.229 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4261.0:1", 2020-03-11T22:55:10.230 INFO:tasks.workunit.client.0.smithi059.stdout: "extra_reqids": [], 2020-03-11T22:55:10.230 INFO:tasks.workunit.client.0.smithi059.stdout: "mtime": "2020-03-11T22:54:51.621677+0000", 2020-03-11T22:55:10.230 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": 0, 2020-03-11T22:55:10.230 INFO:tasks.workunit.client.0.smithi059.stdout: "mod_desc": { 2020-03-11T22:55:10.230 INFO:tasks.workunit.client.0.smithi059.stdout: "object_mod_desc": { 2020-03-11T22:55:10.230 INFO:tasks.workunit.client.0.smithi059.stdout: "can_local_rollback": false, 2020-03-11T22:55:10.230 INFO:tasks.workunit.client.0.smithi059.stdout: "rollback_info_completed": false, 2020-03-11T22:55:10.231 INFO:tasks.workunit.client.0.smithi059.stdout: "ops": [] 2020-03-11T22:55:10.231 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.231 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.231 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_regions": { 2020-03-11T22:55:10.231 INFO:tasks.workunit.client.0.smithi059.stdout: "object_clean_regions": { 2020-03-11T22:55:10.231 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_offsets": "[6900~18446744073709544715]", 2020-03-11T22:55:10.231 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_omap": true, 2020-03-11T22:55:10.232 INFO:tasks.workunit.client.0.smithi059.stdout: "new_object": false 2020-03-11T22:55:10.232 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.232 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.232 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.232 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:55:10.232 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "modify", 2020-03-11T22:55:10.232 INFO:tasks.workunit.client.0.smithi059.stdout: "object": "1:34b5bff6:::obj-26:head", 2020-03-11T22:55:10.233 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'26", 2020-03-11T22:55:10.233 INFO:tasks.workunit.client.0.smithi059.stdout: "prior_version": "0'0", 2020-03-11T22:55:10.233 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4264.0:1", 2020-03-11T22:55:10.233 INFO:tasks.workunit.client.0.smithi059.stdout: "extra_reqids": [], 2020-03-11T22:55:10.233 INFO:tasks.workunit.client.0.smithi059.stdout: "mtime": "2020-03-11T22:54:51.765617+0000", 2020-03-11T22:55:10.233 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": 0, 2020-03-11T22:55:10.233 INFO:tasks.workunit.client.0.smithi059.stdout: "mod_desc": { 2020-03-11T22:55:10.234 INFO:tasks.workunit.client.0.smithi059.stdout: "object_mod_desc": { 2020-03-11T22:55:10.234 INFO:tasks.workunit.client.0.smithi059.stdout: "can_local_rollback": false, 2020-03-11T22:55:10.234 INFO:tasks.workunit.client.0.smithi059.stdout: "rollback_info_completed": false, 2020-03-11T22:55:10.234 INFO:tasks.workunit.client.0.smithi059.stdout: "ops": [] 2020-03-11T22:55:10.234 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.234 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.234 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_regions": { 2020-03-11T22:55:10.235 INFO:tasks.workunit.client.0.smithi059.stdout: "object_clean_regions": { 2020-03-11T22:55:10.235 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_offsets": "[6900~18446744073709544715]", 2020-03-11T22:55:10.235 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_omap": true, 2020-03-11T22:55:10.235 INFO:tasks.workunit.client.0.smithi059.stdout: "new_object": false 2020-03-11T22:55:10.235 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.235 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.235 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.235 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:55:10.236 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "modify", 2020-03-11T22:55:10.236 INFO:tasks.workunit.client.0.smithi059.stdout: "object": "1:a354f8f6:::obj-27:head", 2020-03-11T22:55:10.236 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'27", 2020-03-11T22:55:10.236 INFO:tasks.workunit.client.0.smithi059.stdout: "prior_version": "0'0", 2020-03-11T22:55:10.236 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4267.0:1", 2020-03-11T22:55:10.236 INFO:tasks.workunit.client.0.smithi059.stdout: "extra_reqids": [], 2020-03-11T22:55:10.236 INFO:tasks.workunit.client.0.smithi059.stdout: "mtime": "2020-03-11T22:54:51.800914+0000", 2020-03-11T22:55:10.237 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": 0, 2020-03-11T22:55:10.237 INFO:tasks.workunit.client.0.smithi059.stdout: "mod_desc": { 2020-03-11T22:55:10.237 INFO:tasks.workunit.client.0.smithi059.stdout: "object_mod_desc": { 2020-03-11T22:55:10.237 INFO:tasks.workunit.client.0.smithi059.stdout: "can_local_rollback": false, 2020-03-11T22:55:10.237 INFO:tasks.workunit.client.0.smithi059.stdout: "rollback_info_completed": false, 2020-03-11T22:55:10.237 INFO:tasks.workunit.client.0.smithi059.stdout: "ops": [] 2020-03-11T22:55:10.237 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.238 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.238 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_regions": { 2020-03-11T22:55:10.238 INFO:tasks.workunit.client.0.smithi059.stdout: "object_clean_regions": { 2020-03-11T22:55:10.238 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_offsets": "[6900~18446744073709544715]", 2020-03-11T22:55:10.238 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_omap": true, 2020-03-11T22:55:10.238 INFO:tasks.workunit.client.0.smithi059.stdout: "new_object": false 2020-03-11T22:55:10.238 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.239 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.239 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.239 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:55:10.239 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "modify", 2020-03-11T22:55:10.239 INFO:tasks.workunit.client.0.smithi059.stdout: "object": "1:7421db49:::obj-28:head", 2020-03-11T22:55:10.239 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'28", 2020-03-11T22:55:10.239 INFO:tasks.workunit.client.0.smithi059.stdout: "prior_version": "0'0", 2020-03-11T22:55:10.240 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4270.0:1", 2020-03-11T22:55:10.240 INFO:tasks.workunit.client.0.smithi059.stdout: "extra_reqids": [], 2020-03-11T22:55:10.240 INFO:tasks.workunit.client.0.smithi059.stdout: "mtime": "2020-03-11T22:54:51.836671+0000", 2020-03-11T22:55:10.240 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": 0, 2020-03-11T22:55:10.240 INFO:tasks.workunit.client.0.smithi059.stdout: "mod_desc": { 2020-03-11T22:55:10.240 INFO:tasks.workunit.client.0.smithi059.stdout: "object_mod_desc": { 2020-03-11T22:55:10.240 INFO:tasks.workunit.client.0.smithi059.stdout: "can_local_rollback": false, 2020-03-11T22:55:10.241 INFO:tasks.workunit.client.0.smithi059.stdout: "rollback_info_completed": false, 2020-03-11T22:55:10.241 INFO:tasks.workunit.client.0.smithi059.stdout: "ops": [] 2020-03-11T22:55:10.241 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.241 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.241 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_regions": { 2020-03-11T22:55:10.241 INFO:tasks.workunit.client.0.smithi059.stdout: "object_clean_regions": { 2020-03-11T22:55:10.241 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_offsets": "[6900~18446744073709544715]", 2020-03-11T22:55:10.241 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_omap": true, 2020-03-11T22:55:10.242 INFO:tasks.workunit.client.0.smithi059.stdout: "new_object": false 2020-03-11T22:55:10.242 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.242 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.242 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.242 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:55:10.242 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "modify", 2020-03-11T22:55:10.242 INFO:tasks.workunit.client.0.smithi059.stdout: "object": "1:54bc79ba:::obj-29:head", 2020-03-11T22:55:10.243 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'29", 2020-03-11T22:55:10.243 INFO:tasks.workunit.client.0.smithi059.stdout: "prior_version": "0'0", 2020-03-11T22:55:10.243 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4273.0:1", 2020-03-11T22:55:10.243 INFO:tasks.workunit.client.0.smithi059.stdout: "extra_reqids": [], 2020-03-11T22:55:10.243 INFO:tasks.workunit.client.0.smithi059.stdout: "mtime": "2020-03-11T22:54:51.936716+0000", 2020-03-11T22:55:10.243 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": 0, 2020-03-11T22:55:10.243 INFO:tasks.workunit.client.0.smithi059.stdout: "mod_desc": { 2020-03-11T22:55:10.244 INFO:tasks.workunit.client.0.smithi059.stdout: "object_mod_desc": { 2020-03-11T22:55:10.244 INFO:tasks.workunit.client.0.smithi059.stdout: "can_local_rollback": false, 2020-03-11T22:55:10.244 INFO:tasks.workunit.client.0.smithi059.stdout: "rollback_info_completed": false, 2020-03-11T22:55:10.244 INFO:tasks.workunit.client.0.smithi059.stdout: "ops": [] 2020-03-11T22:55:10.244 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.244 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.244 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_regions": { 2020-03-11T22:55:10.244 INFO:tasks.workunit.client.0.smithi059.stdout: "object_clean_regions": { 2020-03-11T22:55:10.245 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_offsets": "[6900~18446744073709544715]", 2020-03-11T22:55:10.245 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_omap": true, 2020-03-11T22:55:10.245 INFO:tasks.workunit.client.0.smithi059.stdout: "new_object": false 2020-03-11T22:55:10.245 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.245 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.245 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.245 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:55:10.246 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "modify", 2020-03-11T22:55:10.246 INFO:tasks.workunit.client.0.smithi059.stdout: "object": "1:01b15595:::obj-30:head", 2020-03-11T22:55:10.246 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'30", 2020-03-11T22:55:10.246 INFO:tasks.workunit.client.0.smithi059.stdout: "prior_version": "0'0", 2020-03-11T22:55:10.246 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4276.0:1", 2020-03-11T22:55:10.246 INFO:tasks.workunit.client.0.smithi059.stdout: "extra_reqids": [], 2020-03-11T22:55:10.246 INFO:tasks.workunit.client.0.smithi059.stdout: "mtime": "2020-03-11T22:54:52.037234+0000", 2020-03-11T22:55:10.247 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": 0, 2020-03-11T22:55:10.247 INFO:tasks.workunit.client.0.smithi059.stdout: "mod_desc": { 2020-03-11T22:55:10.247 INFO:tasks.workunit.client.0.smithi059.stdout: "object_mod_desc": { 2020-03-11T22:55:10.247 INFO:tasks.workunit.client.0.smithi059.stdout: "can_local_rollback": false, 2020-03-11T22:55:10.247 INFO:tasks.workunit.client.0.smithi059.stdout: "rollback_info_completed": false, 2020-03-11T22:55:10.247 INFO:tasks.workunit.client.0.smithi059.stdout: "ops": [] 2020-03-11T22:55:10.247 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.248 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.248 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_regions": { 2020-03-11T22:55:10.248 INFO:tasks.workunit.client.0.smithi059.stdout: "object_clean_regions": { 2020-03-11T22:55:10.248 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_offsets": "[6900~18446744073709544715]", 2020-03-11T22:55:10.248 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_omap": true, 2020-03-11T22:55:10.248 INFO:tasks.workunit.client.0.smithi059.stdout: "new_object": false 2020-03-11T22:55:10.248 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.248 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.249 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.249 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:55:10.249 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "modify", 2020-03-11T22:55:10.249 INFO:tasks.workunit.client.0.smithi059.stdout: "object": "1:f9a0a8a2:::obj-31:head", 2020-03-11T22:55:10.249 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'31", 2020-03-11T22:55:10.249 INFO:tasks.workunit.client.0.smithi059.stdout: "prior_version": "0'0", 2020-03-11T22:55:10.249 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4279.0:1", 2020-03-11T22:55:10.250 INFO:tasks.workunit.client.0.smithi059.stdout: "extra_reqids": [], 2020-03-11T22:55:10.250 INFO:tasks.workunit.client.0.smithi059.stdout: "mtime": "2020-03-11T22:54:52.076643+0000", 2020-03-11T22:55:10.250 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": 0, 2020-03-11T22:55:10.250 INFO:tasks.workunit.client.0.smithi059.stdout: "mod_desc": { 2020-03-11T22:55:10.250 INFO:tasks.workunit.client.0.smithi059.stdout: "object_mod_desc": { 2020-03-11T22:55:10.250 INFO:tasks.workunit.client.0.smithi059.stdout: "can_local_rollback": false, 2020-03-11T22:55:10.250 INFO:tasks.workunit.client.0.smithi059.stdout: "rollback_info_completed": false, 2020-03-11T22:55:10.251 INFO:tasks.workunit.client.0.smithi059.stdout: "ops": [] 2020-03-11T22:55:10.251 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.251 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.251 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_regions": { 2020-03-11T22:55:10.251 INFO:tasks.workunit.client.0.smithi059.stdout: "object_clean_regions": { 2020-03-11T22:55:10.251 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_offsets": "[6900~18446744073709544715]", 2020-03-11T22:55:10.251 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_omap": true, 2020-03-11T22:55:10.252 INFO:tasks.workunit.client.0.smithi059.stdout: "new_object": false 2020-03-11T22:55:10.252 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.252 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.252 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.252 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:55:10.252 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "modify", 2020-03-11T22:55:10.252 INFO:tasks.workunit.client.0.smithi059.stdout: "object": "1:1bf2562f:::obj-32:head", 2020-03-11T22:55:10.253 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'32", 2020-03-11T22:55:10.253 INFO:tasks.workunit.client.0.smithi059.stdout: "prior_version": "0'0", 2020-03-11T22:55:10.253 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4282.0:1", 2020-03-11T22:55:10.253 INFO:tasks.workunit.client.0.smithi059.stdout: "extra_reqids": [], 2020-03-11T22:55:10.253 INFO:tasks.workunit.client.0.smithi059.stdout: "mtime": "2020-03-11T22:54:52.137179+0000", 2020-03-11T22:55:10.253 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": 0, 2020-03-11T22:55:10.254 INFO:tasks.workunit.client.0.smithi059.stdout: "mod_desc": { 2020-03-11T22:55:10.254 INFO:tasks.workunit.client.0.smithi059.stdout: "object_mod_desc": { 2020-03-11T22:55:10.254 INFO:tasks.workunit.client.0.smithi059.stdout: "can_local_rollback": false, 2020-03-11T22:55:10.254 INFO:tasks.workunit.client.0.smithi059.stdout: "rollback_info_completed": false, 2020-03-11T22:55:10.254 INFO:tasks.workunit.client.0.smithi059.stdout: "ops": [] 2020-03-11T22:55:10.254 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.254 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.255 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_regions": { 2020-03-11T22:55:10.255 INFO:tasks.workunit.client.0.smithi059.stdout: "object_clean_regions": { 2020-03-11T22:55:10.255 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_offsets": "[6900~18446744073709544715]", 2020-03-11T22:55:10.255 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_omap": true, 2020-03-11T22:55:10.255 INFO:tasks.workunit.client.0.smithi059.stdout: "new_object": false 2020-03-11T22:55:10.255 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.255 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.255 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.256 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:55:10.256 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "modify", 2020-03-11T22:55:10.256 INFO:tasks.workunit.client.0.smithi059.stdout: "object": "1:b982ae22:::obj-33:head", 2020-03-11T22:55:10.256 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'33", 2020-03-11T22:55:10.256 INFO:tasks.workunit.client.0.smithi059.stdout: "prior_version": "0'0", 2020-03-11T22:55:10.256 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4285.0:1", 2020-03-11T22:55:10.256 INFO:tasks.workunit.client.0.smithi059.stdout: "extra_reqids": [], 2020-03-11T22:55:10.257 INFO:tasks.workunit.client.0.smithi059.stdout: "mtime": "2020-03-11T22:54:52.221012+0000", 2020-03-11T22:55:10.257 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": 0, 2020-03-11T22:55:10.257 INFO:tasks.workunit.client.0.smithi059.stdout: "mod_desc": { 2020-03-11T22:55:10.257 INFO:tasks.workunit.client.0.smithi059.stdout: "object_mod_desc": { 2020-03-11T22:55:10.257 INFO:tasks.workunit.client.0.smithi059.stdout: "can_local_rollback": false, 2020-03-11T22:55:10.257 INFO:tasks.workunit.client.0.smithi059.stdout: "rollback_info_completed": false, 2020-03-11T22:55:10.257 INFO:tasks.workunit.client.0.smithi059.stdout: "ops": [] 2020-03-11T22:55:10.258 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.258 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.258 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_regions": { 2020-03-11T22:55:10.258 INFO:tasks.workunit.client.0.smithi059.stdout: "object_clean_regions": { 2020-03-11T22:55:10.258 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_offsets": "[6900~18446744073709544715]", 2020-03-11T22:55:10.258 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_omap": true, 2020-03-11T22:55:10.258 INFO:tasks.workunit.client.0.smithi059.stdout: "new_object": false 2020-03-11T22:55:10.258 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.259 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.259 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.259 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:55:10.259 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "modify", 2020-03-11T22:55:10.259 INFO:tasks.workunit.client.0.smithi059.stdout: "object": "1:1ad9f98e:::obj-34:head", 2020-03-11T22:55:10.259 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'34", 2020-03-11T22:55:10.259 INFO:tasks.workunit.client.0.smithi059.stdout: "prior_version": "0'0", 2020-03-11T22:55:10.260 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4288.0:1", 2020-03-11T22:55:10.260 INFO:tasks.workunit.client.0.smithi059.stdout: "extra_reqids": [], 2020-03-11T22:55:10.260 INFO:tasks.workunit.client.0.smithi059.stdout: "mtime": "2020-03-11T22:54:52.288810+0000", 2020-03-11T22:55:10.260 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": 0, 2020-03-11T22:55:10.260 INFO:tasks.workunit.client.0.smithi059.stdout: "mod_desc": { 2020-03-11T22:55:10.260 INFO:tasks.workunit.client.0.smithi059.stdout: "object_mod_desc": { 2020-03-11T22:55:10.260 INFO:tasks.workunit.client.0.smithi059.stdout: "can_local_rollback": false, 2020-03-11T22:55:10.261 INFO:tasks.workunit.client.0.smithi059.stdout: "rollback_info_completed": false, 2020-03-11T22:55:10.261 INFO:tasks.workunit.client.0.smithi059.stdout: "ops": [] 2020-03-11T22:55:10.261 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.261 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.261 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_regions": { 2020-03-11T22:55:10.261 INFO:tasks.workunit.client.0.smithi059.stdout: "object_clean_regions": { 2020-03-11T22:55:10.261 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_offsets": "[6900~18446744073709544715]", 2020-03-11T22:55:10.262 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_omap": true, 2020-03-11T22:55:10.262 INFO:tasks.workunit.client.0.smithi059.stdout: "new_object": false 2020-03-11T22:55:10.262 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.262 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.262 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.262 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:55:10.262 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "modify", 2020-03-11T22:55:10.262 INFO:tasks.workunit.client.0.smithi059.stdout: "object": "1:941f43ff:::obj-35:head", 2020-03-11T22:55:10.263 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'35", 2020-03-11T22:55:10.263 INFO:tasks.workunit.client.0.smithi059.stdout: "prior_version": "0'0", 2020-03-11T22:55:10.263 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4291.0:1", 2020-03-11T22:55:10.263 INFO:tasks.workunit.client.0.smithi059.stdout: "extra_reqids": [], 2020-03-11T22:55:10.263 INFO:tasks.workunit.client.0.smithi059.stdout: "mtime": "2020-03-11T22:54:52.322109+0000", 2020-03-11T22:55:10.263 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": 0, 2020-03-11T22:55:10.263 INFO:tasks.workunit.client.0.smithi059.stdout: "mod_desc": { 2020-03-11T22:55:10.264 INFO:tasks.workunit.client.0.smithi059.stdout: "object_mod_desc": { 2020-03-11T22:55:10.264 INFO:tasks.workunit.client.0.smithi059.stdout: "can_local_rollback": false, 2020-03-11T22:55:10.264 INFO:tasks.workunit.client.0.smithi059.stdout: "rollback_info_completed": false, 2020-03-11T22:55:10.264 INFO:tasks.workunit.client.0.smithi059.stdout: "ops": [] 2020-03-11T22:55:10.264 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.264 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.264 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_regions": { 2020-03-11T22:55:10.264 INFO:tasks.workunit.client.0.smithi059.stdout: "object_clean_regions": { 2020-03-11T22:55:10.265 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_offsets": "[6900~18446744073709544715]", 2020-03-11T22:55:10.265 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_omap": true, 2020-03-11T22:55:10.265 INFO:tasks.workunit.client.0.smithi059.stdout: "new_object": false 2020-03-11T22:55:10.265 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.265 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.265 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.265 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:55:10.266 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "modify", 2020-03-11T22:55:10.266 INFO:tasks.workunit.client.0.smithi059.stdout: "object": "1:24ebabe9:::obj-36:head", 2020-03-11T22:55:10.266 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'36", 2020-03-11T22:55:10.266 INFO:tasks.workunit.client.0.smithi059.stdout: "prior_version": "0'0", 2020-03-11T22:55:10.266 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4294.0:1", 2020-03-11T22:55:10.266 INFO:tasks.workunit.client.0.smithi059.stdout: "extra_reqids": [], 2020-03-11T22:55:10.266 INFO:tasks.workunit.client.0.smithi059.stdout: "mtime": "2020-03-11T22:54:52.396747+0000", 2020-03-11T22:55:10.267 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": 0, 2020-03-11T22:55:10.267 INFO:tasks.workunit.client.0.smithi059.stdout: "mod_desc": { 2020-03-11T22:55:10.267 INFO:tasks.workunit.client.0.smithi059.stdout: "object_mod_desc": { 2020-03-11T22:55:10.267 INFO:tasks.workunit.client.0.smithi059.stdout: "can_local_rollback": false, 2020-03-11T22:55:10.267 INFO:tasks.workunit.client.0.smithi059.stdout: "rollback_info_completed": false, 2020-03-11T22:55:10.267 INFO:tasks.workunit.client.0.smithi059.stdout: "ops": [] 2020-03-11T22:55:10.267 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.267 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.268 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_regions": { 2020-03-11T22:55:10.268 INFO:tasks.workunit.client.0.smithi059.stdout: "object_clean_regions": { 2020-03-11T22:55:10.268 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_offsets": "[6900~18446744073709544715]", 2020-03-11T22:55:10.268 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_omap": true, 2020-03-11T22:55:10.268 INFO:tasks.workunit.client.0.smithi059.stdout: "new_object": false 2020-03-11T22:55:10.268 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.268 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.269 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.269 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:55:10.269 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "modify", 2020-03-11T22:55:10.269 INFO:tasks.workunit.client.0.smithi059.stdout: "object": "1:5f3a5398:::obj-37:head", 2020-03-11T22:55:10.269 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'37", 2020-03-11T22:55:10.269 INFO:tasks.workunit.client.0.smithi059.stdout: "prior_version": "0'0", 2020-03-11T22:55:10.650 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4297.0:1", 2020-03-11T22:55:10.651 INFO:tasks.workunit.client.0.smithi059.stdout: "extra_reqids": [], 2020-03-11T22:55:10.651 INFO:tasks.workunit.client.0.smithi059.stdout: "mtime": "2020-03-11T22:54:52.480552+0000", 2020-03-11T22:55:10.651 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": 0, 2020-03-11T22:55:10.651 INFO:tasks.workunit.client.0.smithi059.stdout: "mod_desc": { 2020-03-11T22:55:10.651 INFO:tasks.workunit.client.0.smithi059.stdout: "object_mod_desc": { 2020-03-11T22:55:10.651 INFO:tasks.workunit.client.0.smithi059.stdout: "can_local_rollback": false, 2020-03-11T22:55:10.652 INFO:tasks.workunit.client.0.smithi059.stdout: "rollback_info_completed": false, 2020-03-11T22:55:10.652 INFO:tasks.workunit.client.0.smithi059.stdout: "ops": [] 2020-03-11T22:55:10.652 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.652 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.652 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_regions": { 2020-03-11T22:55:10.652 INFO:tasks.workunit.client.0.smithi059.stdout: "object_clean_regions": { 2020-03-11T22:55:10.652 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_offsets": "[6900~18446744073709544715]", 2020-03-11T22:55:10.652 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_omap": true, 2020-03-11T22:55:10.653 INFO:tasks.workunit.client.0.smithi059.stdout: "new_object": false 2020-03-11T22:55:10.653 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.653 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.653 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.653 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:55:10.653 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "modify", 2020-03-11T22:55:10.653 INFO:tasks.workunit.client.0.smithi059.stdout: "object": "1:c96cb37e:::obj-38:head", 2020-03-11T22:55:10.654 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'38", 2020-03-11T22:55:10.654 INFO:tasks.workunit.client.0.smithi059.stdout: "prior_version": "0'0", 2020-03-11T22:55:10.654 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4300.0:1", 2020-03-11T22:55:10.654 INFO:tasks.workunit.client.0.smithi059.stdout: "extra_reqids": [], 2020-03-11T22:55:10.654 INFO:tasks.workunit.client.0.smithi059.stdout: "mtime": "2020-03-11T22:54:52.517773+0000", 2020-03-11T22:55:10.654 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": 0, 2020-03-11T22:55:10.654 INFO:tasks.workunit.client.0.smithi059.stdout: "mod_desc": { 2020-03-11T22:55:10.655 INFO:tasks.workunit.client.0.smithi059.stdout: "object_mod_desc": { 2020-03-11T22:55:10.655 INFO:tasks.workunit.client.0.smithi059.stdout: "can_local_rollback": false, 2020-03-11T22:55:10.655 INFO:tasks.workunit.client.0.smithi059.stdout: "rollback_info_completed": false, 2020-03-11T22:55:10.655 INFO:tasks.workunit.client.0.smithi059.stdout: "ops": [] 2020-03-11T22:55:10.655 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.655 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.656 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_regions": { 2020-03-11T22:55:10.656 INFO:tasks.workunit.client.0.smithi059.stdout: "object_clean_regions": { 2020-03-11T22:55:10.656 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_offsets": "[6900~18446744073709544715]", 2020-03-11T22:55:10.656 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_omap": true, 2020-03-11T22:55:10.656 INFO:tasks.workunit.client.0.smithi059.stdout: "new_object": false 2020-03-11T22:55:10.656 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.656 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.657 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.657 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:55:10.657 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "modify", 2020-03-11T22:55:10.657 INFO:tasks.workunit.client.0.smithi059.stdout: "object": "1:69e36996:::obj-39:head", 2020-03-11T22:55:10.657 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'39", 2020-03-11T22:55:10.657 INFO:tasks.workunit.client.0.smithi059.stdout: "prior_version": "0'0", 2020-03-11T22:55:10.657 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4303.0:1", 2020-03-11T22:55:10.658 INFO:tasks.workunit.client.0.smithi059.stdout: "extra_reqids": [], 2020-03-11T22:55:10.658 INFO:tasks.workunit.client.0.smithi059.stdout: "mtime": "2020-03-11T22:54:52.578335+0000", 2020-03-11T22:55:10.658 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": 0, 2020-03-11T22:55:10.658 INFO:tasks.workunit.client.0.smithi059.stdout: "mod_desc": { 2020-03-11T22:55:10.658 INFO:tasks.workunit.client.0.smithi059.stdout: "object_mod_desc": { 2020-03-11T22:55:10.658 INFO:tasks.workunit.client.0.smithi059.stdout: "can_local_rollback": false, 2020-03-11T22:55:10.658 INFO:tasks.workunit.client.0.smithi059.stdout: "rollback_info_completed": false, 2020-03-11T22:55:10.658 INFO:tasks.workunit.client.0.smithi059.stdout: "ops": [] 2020-03-11T22:55:10.659 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.659 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.659 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_regions": { 2020-03-11T22:55:10.659 INFO:tasks.workunit.client.0.smithi059.stdout: "object_clean_regions": { 2020-03-11T22:55:10.659 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_offsets": "[6900~18446744073709544715]", 2020-03-11T22:55:10.659 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_omap": true, 2020-03-11T22:55:10.659 INFO:tasks.workunit.client.0.smithi059.stdout: "new_object": false 2020-03-11T22:55:10.660 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.660 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.660 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.660 INFO:tasks.workunit.client.0.smithi059.stdout: { 2020-03-11T22:55:10.660 INFO:tasks.workunit.client.0.smithi059.stdout: "op": "modify", 2020-03-11T22:55:10.660 INFO:tasks.workunit.client.0.smithi059.stdout: "object": "1:9f75773e:::obj-40:head", 2020-03-11T22:55:10.661 INFO:tasks.workunit.client.0.smithi059.stdout: "version": "34'40", 2020-03-11T22:55:10.661 INFO:tasks.workunit.client.0.smithi059.stdout: "prior_version": "0'0", 2020-03-11T22:55:10.661 INFO:tasks.workunit.client.0.smithi059.stdout: "reqid": "client.4306.0:1", 2020-03-11T22:55:10.661 INFO:tasks.workunit.client.0.smithi059.stdout: "extra_reqids": [], 2020-03-11T22:55:10.661 INFO:tasks.workunit.client.0.smithi059.stdout: "mtime": "2020-03-11T22:54:52.672764+0000", 2020-03-11T22:55:10.661 INFO:tasks.workunit.client.0.smithi059.stdout: "return_code": 0, 2020-03-11T22:55:10.661 INFO:tasks.workunit.client.0.smithi059.stdout: "mod_desc": { 2020-03-11T22:55:10.662 INFO:tasks.workunit.client.0.smithi059.stdout: "object_mod_desc": { 2020-03-11T22:55:10.662 INFO:tasks.workunit.client.0.smithi059.stdout: "can_local_rollback": false, 2020-03-11T22:55:10.662 INFO:tasks.workunit.client.0.smithi059.stdout: "rollback_info_completed": false, 2020-03-11T22:55:10.662 INFO:tasks.workunit.client.0.smithi059.stdout: "ops": [] 2020-03-11T22:55:10.662 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.662 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.662 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_regions": { 2020-03-11T22:55:10.663 INFO:tasks.workunit.client.0.smithi059.stdout: "object_clean_regions": { 2020-03-11T22:55:10.663 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_offsets": "[6900~18446744073709544715]", 2020-03-11T22:55:10.663 INFO:tasks.workunit.client.0.smithi059.stdout: "clean_omap": true, 2020-03-11T22:55:10.663 INFO:tasks.workunit.client.0.smithi059.stdout: "new_object": false 2020-03-11T22:55:10.663 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.663 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.663 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.664 INFO:tasks.workunit.client.0.smithi059.stdout: ], 2020-03-11T22:55:10.664 INFO:tasks.workunit.client.0.smithi059.stdout: "dups": [] 2020-03-11T22:55:10.664 INFO:tasks.workunit.client.0.smithi059.stdout: }, 2020-03-11T22:55:10.664 INFO:tasks.workunit.client.0.smithi059.stdout: "pg_missing_t": { 2020-03-11T22:55:10.664 INFO:tasks.workunit.client.0.smithi059.stdout: "missing": [], 2020-03-11T22:55:10.664 INFO:tasks.workunit.client.0.smithi059.stdout: "may_include_deletes": true 2020-03-11T22:55:10.664 INFO:tasks.workunit.client.0.smithi059.stdout: } 2020-03-11T22:55:10.665 INFO:tasks.workunit.client.0.smithi059.stdout:} 2020-03-11T22:55:10.666 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:82: _common_test: jq '.pg_log_t.log | length' td/osd-backfill-recovery-log/result.log 2020-03-11T22:55:10.666 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:82: _common_test: LOGLEN=40 2020-03-11T22:55:10.666 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:83: _common_test: '[' 40 '!=' 40 ']' 2020-03-11T22:55:10.666 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:87: _common_test: jq '.pg_log_t.dups | length' td/osd-backfill-recovery-log/result.log 2020-03-11T22:55:10.667 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:87: _common_test: DUPSLEN=0 2020-03-11T22:55:10.667 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:88: _common_test: '[' 0 '!=' 0 ']' 2020-03-11T22:55:10.667 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:92: _common_test: grep 'copy_up_to\|copy_after' td/osd-backfill-recovery-log/osd.0.log td/osd-backfill-recovery-log/osd.1.log td/osd-backfill-recovery-log/osd.2.log td/osd-backfill-recovery-log/osd.3.log td/osd-backfill-recovery-log/osd.4.log td/osd-backfill-recovery-log/osd.5.log 2020-03-11T22:55:10.667 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:54.008+0000 7facfdb93700 20 copy_after v 0'0 2020-03-11T22:55:10.667 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:54.008+0000 7facfdb93700 20 copy_after copy log version 34'40 2020-03-11T22:55:10.667 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:54.008+0000 7facfdb93700 20 copy_after copy log version 34'39 2020-03-11T22:55:10.668 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:54.008+0000 7facfdb93700 20 copy_after copy log version 34'38 2020-03-11T22:55:10.668 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:54.008+0000 7facfdb93700 20 copy_after copy log version 34'37 2020-03-11T22:55:10.668 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:54.008+0000 7facfdb93700 20 copy_after copy log version 34'36 2020-03-11T22:55:10.668 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:54.008+0000 7facfdb93700 20 copy_after copy log version 34'35 2020-03-11T22:55:10.668 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:54.008+0000 7facfdb93700 20 copy_after copy log version 34'34 2020-03-11T22:55:10.668 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:54.008+0000 7facfdb93700 20 copy_after copy log version 34'33 2020-03-11T22:55:10.668 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:54.008+0000 7facfdb93700 20 copy_after copy log version 34'32 2020-03-11T22:55:10.668 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:54.008+0000 7facfdb93700 20 copy_after copy log version 34'31 2020-03-11T22:55:10.669 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:54.008+0000 7facfdb93700 20 copy_after copy log version 34'30 2020-03-11T22:55:10.669 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:54.008+0000 7facfdb93700 20 copy_after copy log version 34'29 2020-03-11T22:55:10.669 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:54.008+0000 7facfdb93700 20 copy_after copy log version 34'28 2020-03-11T22:55:10.669 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:54.008+0000 7facfdb93700 20 copy_after copy log version 34'27 2020-03-11T22:55:10.669 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:54.008+0000 7facfdb93700 20 copy_after copy log version 34'26 2020-03-11T22:55:10.669 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:54.008+0000 7facfdb93700 20 copy_after copy log version 34'25 2020-03-11T22:55:10.669 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:54.008+0000 7facfdb93700 20 copy_after copy log version 34'24 2020-03-11T22:55:10.670 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:54.008+0000 7facfdb93700 20 copy_after copy log version 34'23 2020-03-11T22:55:10.670 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:54.008+0000 7facfdb93700 20 copy_after copy log version 34'22 2020-03-11T22:55:10.670 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:54.008+0000 7facfdb93700 20 copy_after copy log version 34'21 2020-03-11T22:55:10.670 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:54.008+0000 7facfdb93700 20 copy_after copy log version 34'20 2020-03-11T22:55:10.670 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:54.008+0000 7facfdb93700 20 copy_after copy log version 34'19 2020-03-11T22:55:10.670 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:54.008+0000 7facfdb93700 20 copy_after copy log version 34'18 2020-03-11T22:55:10.670 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:54.008+0000 7facfdb93700 20 copy_after copy log version 34'17 2020-03-11T22:55:10.671 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:54.008+0000 7facfdb93700 20 copy_after copy log version 34'16 2020-03-11T22:55:10.671 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:54.008+0000 7facfdb93700 20 copy_after copy log version 34'15 2020-03-11T22:55:10.671 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:54.008+0000 7facfdb93700 20 copy_after copy log version 34'14 2020-03-11T22:55:10.671 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:54.008+0000 7facfdb93700 20 copy_after copy log version 34'13 2020-03-11T22:55:10.671 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:54.008+0000 7facfdb93700 20 copy_after copy log version 34'12 2020-03-11T22:55:10.672 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:54.008+0000 7facfdb93700 20 copy_after copy log version 34'11 2020-03-11T22:55:10.672 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:54.008+0000 7facfdb93700 20 copy_after copy log version 34'10 2020-03-11T22:55:10.672 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:54.008+0000 7facfdb93700 20 copy_after copy log version 34'9 2020-03-11T22:55:10.672 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:54.008+0000 7facfdb93700 20 copy_after copy log version 34'8 2020-03-11T22:55:10.672 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:54.008+0000 7facfdb93700 20 copy_after copy log version 34'7 2020-03-11T22:55:10.672 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:54.008+0000 7facfdb93700 20 copy_after copy log version 34'6 2020-03-11T22:55:10.672 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:54.008+0000 7facfdb93700 20 copy_after copy log version 34'5 2020-03-11T22:55:10.673 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:54.008+0000 7facfdb93700 20 copy_after copy log version 34'4 2020-03-11T22:55:10.673 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:54.008+0000 7facfdb93700 20 copy_after copy log version 34'3 2020-03-11T22:55:10.673 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:54.008+0000 7facfdb93700 20 copy_after copy log version 34'2 2020-03-11T22:55:10.673 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:54.008+0000 7facfdb93700 20 copy_after copy log version 34'1 2020-03-11T22:55:10.673 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:54.008+0000 7facfdb93700 20 copy_up_to/copy_after earliest_dup_version 0 2020-03-11T22:55:10.673 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after v 0'0 2020-03-11T22:55:10.674 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'40 2020-03-11T22:55:10.674 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'39 2020-03-11T22:55:10.674 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'38 2020-03-11T22:55:10.674 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'37 2020-03-11T22:55:10.674 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'36 2020-03-11T22:55:10.674 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'35 2020-03-11T22:55:10.674 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'34 2020-03-11T22:55:10.675 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'33 2020-03-11T22:55:10.675 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'32 2020-03-11T22:55:10.675 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'31 2020-03-11T22:55:10.675 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'30 2020-03-11T22:55:10.675 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'29 2020-03-11T22:55:10.675 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'28 2020-03-11T22:55:10.675 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'27 2020-03-11T22:55:10.676 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'26 2020-03-11T22:55:10.676 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'25 2020-03-11T22:55:10.676 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'24 2020-03-11T22:55:10.676 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'23 2020-03-11T22:55:10.676 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'22 2020-03-11T22:55:10.676 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'21 2020-03-11T22:55:10.676 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'20 2020-03-11T22:55:10.677 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'19 2020-03-11T22:55:10.677 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'18 2020-03-11T22:55:10.677 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'17 2020-03-11T22:55:10.677 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'16 2020-03-11T22:55:10.677 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'15 2020-03-11T22:55:10.677 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'14 2020-03-11T22:55:10.677 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'13 2020-03-11T22:55:10.678 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'12 2020-03-11T22:55:10.678 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'11 2020-03-11T22:55:10.678 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'10 2020-03-11T22:55:10.678 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'9 2020-03-11T22:55:10.678 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'8 2020-03-11T22:55:10.678 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'7 2020-03-11T22:55:10.678 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'6 2020-03-11T22:55:10.679 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'5 2020-03-11T22:55:10.679 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'4 2020-03-11T22:55:10.679 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'3 2020-03-11T22:55:10.679 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'2 2020-03-11T22:55:10.679 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'1 2020-03-11T22:55:10.679 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_up_to/copy_after earliest_dup_version 0 2020-03-11T22:55:10.679 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after v 0'0 2020-03-11T22:55:10.680 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'40 2020-03-11T22:55:10.680 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'39 2020-03-11T22:55:10.680 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'38 2020-03-11T22:55:10.680 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'37 2020-03-11T22:55:10.680 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'36 2020-03-11T22:55:10.680 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'35 2020-03-11T22:55:10.680 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'34 2020-03-11T22:55:10.680 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'33 2020-03-11T22:55:10.681 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'32 2020-03-11T22:55:10.681 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'31 2020-03-11T22:55:10.681 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'30 2020-03-11T22:55:10.681 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'29 2020-03-11T22:55:10.681 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'28 2020-03-11T22:55:10.681 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'27 2020-03-11T22:55:10.682 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'26 2020-03-11T22:55:10.682 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'25 2020-03-11T22:55:10.682 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'24 2020-03-11T22:55:10.682 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'23 2020-03-11T22:55:10.682 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'22 2020-03-11T22:55:10.685 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'21 2020-03-11T22:55:10.685 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'20 2020-03-11T22:55:10.685 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'19 2020-03-11T22:55:10.685 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'18 2020-03-11T22:55:10.685 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'17 2020-03-11T22:55:10.686 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'16 2020-03-11T22:55:10.686 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'15 2020-03-11T22:55:10.686 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'14 2020-03-11T22:55:10.686 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'13 2020-03-11T22:55:10.686 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'12 2020-03-11T22:55:10.686 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'11 2020-03-11T22:55:10.687 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'10 2020-03-11T22:55:10.687 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'9 2020-03-11T22:55:10.687 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'8 2020-03-11T22:55:10.687 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'7 2020-03-11T22:55:10.687 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'6 2020-03-11T22:55:10.687 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'5 2020-03-11T22:55:10.687 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'4 2020-03-11T22:55:10.687 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'3 2020-03-11T22:55:10.688 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'2 2020-03-11T22:55:10.688 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_after copy log version 34'1 2020-03-11T22:55:10.688 INFO:tasks.workunit.client.0.smithi059.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-03-11T22:54:54.102+0000 7fd654bc5700 20 copy_up_to/copy_after earliest_dup_version 0 2020-03-11T22:55:10.688 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:93: _common_test: rm -f td/osd-backfill-recovery-log/result.log 2020-03-11T22:55:10.688 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:94: _common_test: '[' 0 '!=' 0 ']' 2020-03-11T22:55:10.688 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:34: run: teardown td/osd-backfill-recovery-log 2020-03-11T22:55:10.689 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-recovery-log 2020-03-11T22:55:10.689 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:55:10.689 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-recovery-log KILL 2020-03-11T22:55:10.689 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:55:10.689 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:55:10.689 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:55:10.689 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:55:10.690 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:55:10.690 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:55:10.690 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:55:10.690 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:55:10.690 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:55:10.690 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:55:10.690 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:55:10.691 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:55:10.691 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:55:10.691 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:55:10.691 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:55:10.691 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:55:10.691 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:55:10.692 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:55:10.692 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-recovery-log 2020-03-11T22:55:10.694 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:55:10.694 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:55:10.694 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76608 2020-03-11T22:55:10.694 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76608 2020-03-11T22:55:10.696 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:55:10.696 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:55:10.696 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:31: run: for func in $funcs 2020-03-11T22:55:10.697 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:32: run: setup td/osd-backfill-recovery-log 2020-03-11T22:55:10.697 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-recovery-log 2020-03-11T22:55:10.697 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-recovery-log 2020-03-11T22:55:10.697 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-recovery-log 2020-03-11T22:55:10.697 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:55:10.697 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-recovery-log KILL 2020-03-11T22:55:10.698 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:55:10.698 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:55:10.698 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:55:10.698 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:55:10.698 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:55:10.699 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:55:10.699 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:55:10.699 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:55:10.700 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:55:10.700 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:55:10.700 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:55:10.701 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:55:10.702 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:55:10.702 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:55:10.703 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:55:10.703 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:55:10.704 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:55:10.705 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:55:10.705 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-recovery-log 2020-03-11T22:55:10.706 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:55:10.706 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:55:10.706 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76608 2020-03-11T22:55:10.706 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76608 2020-03-11T22:55:10.707 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:55:10.707 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:55:10.707 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-recovery-log 2020-03-11T22:55:10.708 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-11T22:55:10.708 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:55:10.709 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76608 2020-03-11T22:55:10.709 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76608 2020-03-11T22:55:10.712 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-11T22:55:10.712 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-03-11T22:55:10.712 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-11T22:55:10.712 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-recovery-log 1' TERM HUP INT 2020-03-11T22:55:10.712 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:33: run: TEST_recovery_2 td/osd-backfill-recovery-log 2020-03-11T22:55:10.712 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:127: TEST_recovery_2: local dir=td/osd-backfill-recovery-log 2020-03-11T22:55:10.712 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:129: TEST_recovery_2: _common_test td/osd-backfill-recovery-log '--osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10' 151 10 141 20 2020-03-11T22:55:10.713 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:40: _common_test: local dir=td/osd-backfill-recovery-log 2020-03-11T22:55:10.713 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:41: _common_test: local 'extra_opts=--osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10' 2020-03-11T22:55:10.713 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:42: _common_test: local loglen=151 2020-03-11T22:55:10.713 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:43: _common_test: local dupslen=10 2020-03-11T22:55:10.713 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:44: _common_test: local objects=141 2020-03-11T22:55:10.713 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:45: _common_test: local moreobjects=20 2020-03-11T22:55:10.714 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:47: _common_test: local OSDS=6 2020-03-11T22:55:10.714 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:49: _common_test: run_mon td/osd-backfill-recovery-log a 2020-03-11T22:55:10.714 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-recovery-log 2020-03-11T22:55:10.714 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-11T22:55:10.714 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-11T22:55:10.714 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-11T22:55:10.714 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-recovery-log/a 2020-03-11T22:55:10.715 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-recovery-log/a --run-dir=td/osd-backfill-recovery-log 2020-03-11T22:55:11.033 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-11T22:55:11.034 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:55:11.034 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:55:11.034 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:55:11.034 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:55:11.034 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76608 2020-03-11T22:55:11.034 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76608/$cluster-$name.asok' 2020-03-11T22:55:11.035 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-recovery-log/a '--log-file=td/osd-backfill-recovery-log/$name.log' '--admin-socket=/tmp/ceph-asok.76608/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-recovery-log/log --run-dir=td/osd-backfill-recovery-log '--pid-file=td/osd-backfill-recovery-log/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-11T22:55:11.242 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-11T22:55:11.243 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-11T22:55:11.243 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:55:11.243 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:55:11.243 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-11T22:55:11.245 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-11T22:55:11.245 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:55:11.245 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:55:11.245 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:55:11.245 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:55:11.246 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:55:11.246 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76608 2020-03-11T22:55:11.246 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76608/ceph-mon.a.asok 2020-03-11T22:55:11.246 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:55:11.246 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.76608/ceph-mon.a.asok config get fsid 2020-03-11T22:55:11.346 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-11T22:55:11.347 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:55:11.347 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:55:11.347 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-11T22:55:11.347 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-11T22:55:11.347 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:55:11.347 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:55:11.348 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:55:11.348 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:55:11.348 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:55:11.348 INFO:tasks.workunit.client.0.smithi059.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76608 2020-03-11T22:55:11.348 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76608/ceph-mon.a.asok 2020-03-11T22:55:11.348 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:55:11.349 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.76608/ceph-mon.a.asok config get mon_host 2020-03-11T22:55:11.439 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:50: _common_test: run_mgr td/osd-backfill-recovery-log x 2020-03-11T22:55:11.439 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/osd-backfill-recovery-log 2020-03-11T22:55:11.440 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-11T22:55:11.440 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-11T22:55:11.440 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-11T22:55:11.440 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/osd-backfill-recovery-log/x 2020-03-11T22:55:11.440 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-11T22:55:12.371 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-11T22:55:12.371 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:55:12.371 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:55:12.371 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:55:12.371 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:55:12.371 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76608 2020-03-11T22:55:12.372 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76608/$cluster-$name.asok' 2020-03-11T22:55:12.372 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:55:12.372 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-recovery-log/x '--log-file=td/osd-backfill-recovery-log/$name.log' '--admin-socket=/tmp/ceph-asok.76608/$cluster-$name.asok' --run-dir=td/osd-backfill-recovery-log '--pid-file=td/osd-backfill-recovery-log/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:55:12.396 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:51: _common_test: export CEPH_ARGS 2020-03-11T22:55:12.397 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:52: _common_test: export 'EXTRA_OPTS= --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10' 2020-03-11T22:55:12.397 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:52: _common_test: EXTRA_OPTS=' --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10' 2020-03-11T22:55:12.397 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: expr 6 - 1 2020-03-11T22:55:12.398 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: seq 0 5 2020-03-11T22:55:12.402 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:55:12.402 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 0 2020-03-11T22:55:12.403 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:55:12.403 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:55:12.403 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-11T22:55:12.403 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:55:12.403 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/0 2020-03-11T22:55:12.403 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=7f0bc282-eebc-452f-ae5f-6537699866e7 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:55:12.404 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:55:12.404 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:55:12.404 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:55:12.404 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/0' 2020-03-11T22:55:12.404 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/0/journal' 2020-03-11T22:55:12.405 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:55:12.405 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10' 2020-03-11T22:55:12.405 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:55:12.405 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:55:12.405 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:55:12.405 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:55:12.405 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:55:12.406 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:55:12.406 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76608 2020-03-11T22:55:12.406 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76608/$cluster-$name.asok' 2020-03-11T22:55:12.406 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76608/$cluster-$name.asok' 2020-03-11T22:55:12.406 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:55:12.406 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:55:12.406 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:55:12.407 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:55:12.407 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:55:12.407 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:55:12.407 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:55:12.407 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:55:12.407 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:55:12.408 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:55:12.408 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/0 2020-03-11T22:55:12.408 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:55:12.408 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=4ddac105-ac59-42d3-bff7-2c9d2571fe11 2020-03-11T22:55:12.408 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 4ddac105-ac59-42d3-bff7-2c9d2571fe11' 2020-03-11T22:55:12.408 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:55:12.408 INFO:tasks.workunit.client.0.smithi059.stdout:add osd0 4ddac105-ac59-42d3-bff7-2c9d2571fe11 2020-03-11T22:55:12.416 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBQbGleU3eqGBAA1PWTY/Ylo2G+3Fk3ro01aA== 2020-03-11T22:55:12.417 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBQbGleU3eqGBAA1PWTY/Ylo2G+3Fk3ro01aA=="}' 2020-03-11T22:55:12.417 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 4ddac105-ac59-42d3-bff7-2c9d2571fe11 -i td/osd-backfill-recovery-log/0/new.json 2020-03-11T22:55:12.752 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:55:12.761 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/0/new.json 2020-03-11T22:55:12.762 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=7f0bc282-eebc-452f-ae5f-6537699866e7 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/0 --osd-journal=td/osd-backfill-recovery-log/0/journal --chdir= --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76608/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBQbGleU3eqGBAA1PWTY/Ylo2G+3Fk3ro01aA== --osd-uuid 4ddac105-ac59-42d3-bff7-2c9d2571fe11 2020-03-11T22:55:12.789 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:55:12.785+0000 7f940dc7fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:12.790 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:55:12.787+0000 7f940dc7fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:12.792 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:55:12.788+0000 7f940dc7fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:12.792 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:55:12.789+0000 7f940dc7fec0 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2020-03-11T22:55:12.792 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:55:12.789+0000 7f940dc7fec0 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2020-03-11T22:55:12.792 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:55:12.789+0000 7f940dc7fec0 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2020-03-11T22:55:12.883 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:55:12.880+0000 7f940dc7fec0 -1 bluestore(td/osd-backfill-recovery-log/0) _read_fsid unparsable uuid 2020-03-11T22:55:16.006 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/0/keyring 2020-03-11T22:55:16.006 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:55:16.007 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-11T22:55:16.007 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:55:16.007 INFO:tasks.workunit.client.0.smithi059.stdout:adding osd0 key to auth repository 2020-03-11T22:55:16.492 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-11T22:55:16.492 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:55:16.492 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:55:16.493 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:55:16.493 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=7f0bc282-eebc-452f-ae5f-6537699866e7 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/0 --osd-journal=td/osd-backfill-recovery-log/0/journal --chdir= --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76608/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:55:16.493 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.0 2020-03-11T22:55:16.515 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:55:16.512+0000 7efc0262eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:16.517 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:55:16.514+0000 7efc0262eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:16.519 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:55:16.515+0000 7efc0262eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:16.863 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-11T22:55:16.863 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:55:16.863 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:55:16.863 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:55:16.863 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:55:16.863 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:55:16.864 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:55:16.864 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:55:16.864 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:55:16.864 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:55:17.043 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:55:17.040+0000 7efc0262eec0 -1 Falling back to public interface 2020-03-11T22:55:17.225 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:55:17.816 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:55:17.813+0000 7efc0262eec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-11T22:55:18.226 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:55:18.227 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:55:18.227 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:55:18.227 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:55:18.227 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:55:18.227 INFO:tasks.workunit.client.0.smithi059.stdout:1 2020-03-11T22:55:18.609 INFO:tasks.workunit.client.0.smithi059.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/97739,v1:127.0.0.1:6803/97739] [v2:127.0.0.1:6804/97739,v1:127.0.0.1:6805/97739] exists,up 4ddac105-ac59-42d3-bff7-2c9d2571fe11 2020-03-11T22:55:18.610 INFO:tasks.workunit.client.0.smithi059.stdout:add osd1 7df20859-9422-43ba-a47f-7e2b93c3d590 2020-03-11T22:55:18.610 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:55:18.610 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:55:18.610 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:55:18.611 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:55:18.611 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 1 2020-03-11T22:55:18.611 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:55:18.611 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:55:18.612 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-11T22:55:18.612 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:55:18.612 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/1 2020-03-11T22:55:18.612 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=7f0bc282-eebc-452f-ae5f-6537699866e7 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:55:18.612 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:55:18.612 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:55:18.612 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:55:18.613 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/1' 2020-03-11T22:55:18.613 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/1/journal' 2020-03-11T22:55:18.613 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:55:18.613 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10' 2020-03-11T22:55:18.613 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:55:18.614 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:55:18.614 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:55:18.614 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:55:18.614 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:55:18.614 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:55:18.614 INFO:tasks.workunit.client.0.smithi059.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76608 2020-03-11T22:55:18.615 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76608/$cluster-$name.asok' 2020-03-11T22:55:18.615 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76608/$cluster-$name.asok' 2020-03-11T22:55:18.615 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:55:18.615 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:55:18.616 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:55:18.616 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:55:18.616 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:55:18.616 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:55:18.616 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:55:18.616 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:55:18.617 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:55:18.617 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:55:18.617 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/1 2020-03-11T22:55:18.617 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:55:18.617 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=7df20859-9422-43ba-a47f-7e2b93c3d590 2020-03-11T22:55:18.617 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 7df20859-9422-43ba-a47f-7e2b93c3d590' 2020-03-11T22:55:18.618 INFO:tasks.workunit.client.0.smithi059.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:55:18.623 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBWbGlell73JBAA4OBvY06rUennNMmHni1qMQ== 2020-03-11T22:55:18.623 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBWbGlell73JBAA4OBvY06rUennNMmHni1qMQ=="}' 2020-03-11T22:55:18.623 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 7df20859-9422-43ba-a47f-7e2b93c3d590 -i td/osd-backfill-recovery-log/1/new.json 2020-03-11T22:55:19.119 INFO:tasks.workunit.client.0.smithi059.stdout:1 2020-03-11T22:55:19.129 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/1/new.json 2020-03-11T22:55:19.130 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=7f0bc282-eebc-452f-ae5f-6537699866e7 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/1 --osd-journal=td/osd-backfill-recovery-log/1/journal --chdir= --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76608/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBWbGlell73JBAA4OBvY06rUennNMmHni1qMQ== --osd-uuid 7df20859-9422-43ba-a47f-7e2b93c3d590 2020-03-11T22:55:19.155 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:55:19.152+0000 7f6f615a5ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:19.189 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:55:19.186+0000 7f6f615a5ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:19.190 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:55:19.187+0000 7f6f615a5ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:19.191 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:55:19.188+0000 7f6f615a5ec0 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2020-03-11T22:55:19.191 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:55:19.188+0000 7f6f615a5ec0 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2020-03-11T22:55:19.191 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:55:19.188+0000 7f6f615a5ec0 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2020-03-11T22:55:19.255 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:55:19.252+0000 7f6f615a5ec0 -1 bluestore(td/osd-backfill-recovery-log/1) _read_fsid unparsable uuid 2020-03-11T22:55:22.673 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/1/keyring 2020-03-11T22:55:22.673 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:55:22.674 INFO:tasks.workunit.client.0.smithi059.stdout:adding osd1 key to auth repository 2020-03-11T22:55:22.674 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-03-11T22:55:22.675 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:55:23.154 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-03-11T22:55:23.154 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=7f0bc282-eebc-452f-ae5f-6537699866e7 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/1 --osd-journal=td/osd-backfill-recovery-log/1/journal --chdir= --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76608/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:55:23.155 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:55:23.155 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:55:23.155 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:55:23.155 INFO:tasks.workunit.client.0.smithi059.stdout:start osd.1 2020-03-11T22:55:23.177 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:55:23.174+0000 7fdd8c605ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:23.179 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:55:23.176+0000 7fdd8c605ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:23.180 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:55:23.177+0000 7fdd8c605ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:23.537 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-03-11T22:55:23.537 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:55:23.537 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:55:23.537 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:55:23.538 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:55:23.538 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:55:23.538 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:55:23.538 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:55:23.539 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:55:23.539 INFO:tasks.workunit.client.0.smithi059.stdout:0 2020-03-11T22:55:23.702 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:55:23.699+0000 7fdd8c605ec0 -1 Falling back to public interface 2020-03-11T22:55:23.900 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:55:24.440 INFO:tasks.workunit.client.0.smithi059.stderr:2020-03-11T22:55:24.437+0000 7fdd8c605ec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-11T22:55:24.902 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:55:24.902 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:55:24.902 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:55:24.902 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:55:24.902 INFO:tasks.workunit.client.0.smithi059.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:55:24.902 INFO:tasks.workunit.client.0.smithi059.stdout:1 2020-03-11T22:55:24.989 DEBUG:teuthology.exit:Got signal 15; running 2 handlers... 2020-03-11T22:55:24.989 DEBUG:teuthology.task.console_log:Killing console logger for smithi059 2020-03-11T22:55:24.990 DEBUG:teuthology.task.console_log:Killing console logger for smithi059 2020-03-11T22:55:24.990 DEBUG:teuthology.exit:Finished running handlers